What is Tokenization?

Tokenisation splits text into the units a model actually processes. For language models these are subword tokens, so token count does not match word count.

2 tools that work with Tokenization

Files never leave your browser.

AI & LLM Tools2

LLM Token CounterCount tokens with the real BPE vocabulary, see every token coloured in place, and compare the count across models.Words to Tokens ConverterConvert between words, characters, pages and tokens using measured ratios for prose, code, JSON and CJK.

Related terms

Terms that appear alongside Tokenization on the same tools.

← All glossary terms Browse every tool →