Module tokens 

Token counting using OpenAI-compatible cl100k_base BPE tokenizer.

All providers supported by tiny-agent-core (DeepSeek, Kimi, etc.) use tokenizers that are highly compatible with cl100k_base. This module provides a shared, lazily-loaded counter that replaces the old 4-char-per- token heuristic with an accurate (±5%) token count.

Functions

count_tokens

Count tokens in text using the cl100k_base tokenizer.