Function count_tokens 

pub fn count_tokens(text: &str) -> usize

Count tokens in text using the cl100k_base tokenizer.

If the tokenizer fails to load (e.g. corrupted vocabulary file), falls back to the legacy heuristic: ceil(text.len() / 4).