Sampling
The dice roll. The AI has a list of possible next tokens with probabilities (the distribution). Sampling is the step where it picks one. This is why you ask the same question twice and get two different answers. The probabilities didn't change. The sample did.
Sampling is the stochastic process of selecting a single token from the model's output probability distribution at each generation step. By introducing randomness into the selection, the model avoids deterministic repetition. This is why identical prompts can yield different outputs; the underlying distribution remains static, but the specific sample drawn from that distribution varies.
Sampling is the discrete selection mechanism applied to the model's output logit distribution (post-softmax) to determine the next token in a sequence. It functions as a stochastic mapping from a probability distribution to a single token index. Strategies such as greedy decoding, top-k, and nucleus (top-p) sampling define the constraints of this selection. The variance in generation across identical prompts is a direct result of the sampling step, as the probability distribution remains invariant while the realization of the random variable changes.
evolution
- 1950 · historyTuring Test and Probabilistic Foundations
Alan Turing's 'Computing Machinery and Intelligence' established the conceptual framework for machines making non-deterministic, human-like choices.
- 1980 · historyMarkov Chain Monte Carlo (MCMC)
The formalization of MCMC methods allowed researchers to sample from complex probability distributions, a foundational technique for modern generative models.
- 2014 · historyGenerative Adversarial Networks (GANs)
Ian Goodfellow introduced GANs, which rely on sampling from a latent space to generate new data, shifting focus toward stochastic output generation.
- 2017 · historyTransformer Architecture
The 'Attention Is All You Need' paper introduced the softmax output layer, standardizing the probability distribution over tokens that modern sampling methods now navigate.
- 2020 · historyTop-p (Nucleus) Sampling
Holtzman et al. introduced nucleus sampling to solve the 'degeneration' problem in text generation by dynamically truncating the tail of the probability distribution.