Expand description
Versioned token accounting (Definition 4’s size function; Definition 6’s
accountant c).
v1 is a conservative upper bound, not an exact tokenizer: BPE English
and code average roughly four bytes per token, so ceil(bytes / 3)
overestimates by a comfortable margin, and Definition 6’s guard reserve
absorbs residual error. An underestimating accountant is not valid. The
accountant’s identity and mode enter every program digest, so a later
exact tokenizer is a clean versioned swap with no silent replay
divergence.
Structs§
- Token
Accountant Ref - A named, versioned token accounting function.
Enums§
- Accounting
Mode - Whether counts are exact or a declared upper bound.