Tokenisation
The step before inference — your text gets chopped into tokens the model recognises. Because the model doesn't read words or letters, it must first translate your input into this numeric format.
Tokenisation is the preprocessing step where raw input text is segmented into discrete units called tokens, which are then mapped to unique integer IDs from a model's vocabulary; this numeric representation is the mandatory format required for the model to perform inference.
Tokenisation is the deterministic mapping of a character sequence into a sequence of discrete tokens via a specific algorithm (e.g., Byte Pair Encoding or WordPiece), which are subsequently indexed into a fixed-size vocabulary; this transformation into integer embeddings is a prerequisite for the model's input layer to process the data through its neural architecture.
evolution
- 1960 · historyEarly Lexical Analysis
The concept of tokenization emerged in early compiler design to convert source code strings into meaningful symbols for parsing.
- 2016 · historyByte Pair Encoding (BPE)
Sennrich et al. introduced BPE to NLP, allowing models to handle rare words by breaking them into subword units.
- 2017 · historyTransformer Tokenization
The 'Attention Is All You Need' paper popularized fixed-vocabulary subword tokenization as the standard input method for Transformer architectures.
- 2020 · historyByte-level BPE
GPT-2 and GPT-3 popularized byte-level BPE, which eliminates out-of-vocabulary issues by tokenizing at the byte level rather than the character level.
- 2023 · historyToken-free Architectures
Emerging research into 'token-free' models like CANINE and ByT5 began exploring direct processing of raw bytes or characters to bypass traditional tokenization bottlenecks.