Skip to main content

Module accountant

Module accountant 

Source
Expand description

Versioned token accounting (Definition 4’s size function; Definition 6’s accountant c).

v1 is a conservative upper bound, not an exact tokenizer: BPE English and code average roughly four bytes per token, so ceil(bytes / 3) overestimates by a comfortable margin, and Definition 6’s guard reserve absorbs residual error. An underestimating accountant is not valid. The accountant’s identity and mode enter every program digest, so a later exact tokenizer is a clean versioned swap with no silent replay divergence.

Structs§

TokenAccountantRef
A named, versioned token accounting function.

Enums§

AccountingMode
Whether counts are exact or a declared upper bound.