SIGNAL//DESK
AI/MLsrc: Roost FA glossary v2

Probability distribution

At every word, the AI computes a list: every possible next token, with a probability beside it. 'the' might be 23%, 'a' 18%, 'your' 4%, ... down to 0.0001% for unlikely tokens. THAT list is the probability distribution. Then it picks one. Then it does the next word. Then the next.

At each inference step, the model generates a probability distribution—a vector representing the likelihood of every token in its vocabulary being the next output. This distribution is calculated by applying a softmax function to the model's final hidden state, ensuring all values sum to 1.0. The sampler then uses this distribution to select the next token before the process repeats.

The probability distribution is a discrete probability mass function over the model's entire vocabulary, computed as the output of the final softmax layer in the transformer architecture. For a vocabulary of size N, it is a vector of N non-negative values that sum to 1.0, where each element represents the conditional probability of a specific token given the preceding context. This vector serves as the input to the sampling algorithm, which selects the next token to be appended to the sequence before the autoregressive loop proceeds to the next step.

evolution

  1. 1948 · history
    Information Theory

    Claude Shannon introduces entropy and the concept of probability distributions in communication, forming the mathematical foundation for language modeling.

  2. 1986 · history
    Connectionist Language Models

    Rumelhart, Hinton, and Williams popularize backpropagation, enabling neural networks to learn probability distributions over sequences.

  3. 2003 · history
    Neural Probabilistic Language Models

    Yoshua Bengio et al. introduce the first neural language model that learns a distributed representation of words to predict the next token via a softmax probability distribution.

  4. 2017 · history
    The Transformer Architecture

    Vaswani et al. introduce the Transformer, which uses self-attention to compute highly accurate probability distributions for next-token prediction at scale.

← all terms