What is Byte pair encoding?

Byte pair encoding builds a subword vocabulary by repeatedly merging the most frequent adjacent pair of symbols in a corpus. It is the tokenisation scheme behind most current language models.

2 tools that work with Byte pair encoding

Files never leave your browser.

AI & LLM Tools2

LLM Token CounterCount tokens with the real BPE vocabulary, see every token coloured in place, and compare the count across models.Words to Tokens ConverterConvert between words, characters, pages and tokens using measured ratios for prose, code, JSON and CJK.

Related terms

Terms that appear alongside Byte pair encoding on the same tools.

← All glossary terms Browse every tool →