Token
The piece the AI reads in. It's not letters — it's chunks. The word 'unbelievable' might be three tokens. 'Mr Tan' might be one token plus another. This is why prompts and replies have character/word counts that feel weird.
A token is the fundamental unit of input and output for an LLM, representing a sequence of characters that the model processes as a single entity. Since tokenization varies by model architecture, a single word might be split into multiple tokens or grouped into one, which explains why character and word counts do not align perfectly with token-based usage limits.
A token is the atomic unit of text representation within an LLM's vocabulary, generated by a tokenizer that maps raw strings into integer indices. These units are not strictly isomorphic to words or characters; they are sub-word sequences determined by the model's specific byte-pair encoding or similar statistical compression scheme. Consequently, token-based constraints are the primary metric for context window capacity and computational cost, often diverging from human-centric linguistic metrics.
evolution
- 1994 · historyByte Pair Encoding (BPE) Introduction
Philip Gage introduced BPE as a data compression algorithm, which later became the foundation for modern subword tokenization.
- 2016 · historyBPE for Neural Machine Translation
Sennrich et al. popularized BPE as a method to handle rare words in neural machine translation, effectively solving the out-of-vocabulary problem.
- 2018 · historyWordPiece and SentencePiece
Google and others refined subword tokenization with models like BERT, standardizing the use of tokens as the primary unit of input for Large Language Models.
- 2020 · historyGPT-3 Tokenization
OpenAI implemented a robust byte-level BPE tokenizer, establishing the current industry standard for how LLMs process text into numerical sequences.