AI/MLsrc: Roost FA glossary v2

Inference

Running the model to get an answer — the live use of a trained model. Every time you hit "send" in Roost, you are triggering an inference cycle.

Inference is the operational phase of a machine learning model where input data is processed through the trained weights to produce a prediction. In the context of LLMs, this is the cycle triggered when a user submits a prompt, resulting in the generation of new tokens based on the model's learned parameters.

Inference is the computational execution of a trained model's forward pass, wherein input data is transformed through a series of mathematical operations—typically matrix multiplications and non-linear activations—to derive an output. It represents the active deployment phase where the model calculates conditional probability distributions over a vocabulary to predict subsequent tokens, effectively mapping input sequences to output sequences without updating the model's internal weights.

evolution

1950 · history
Turing Test Proposal
Alan Turing formalizes the concept of machine intelligence evaluation, establishing the foundational requirement for an agent to produce human-like responses.
1980 · history
Expert Systems Era
The rise of rule-based systems introduced the first practical 'inference engines' that applied logical rules to knowledge bases to derive conclusions.
2012 · history
Deep Learning Breakthrough
The success of AlexNet at the ImageNet competition shifted the paradigm toward neural network inference, where models process data through learned weights.
2017 · history
Transformer Architecture
The introduction of the Transformer model enabled highly parallelizable inference, allowing for the rapid, large-scale processing required by modern LLMs.
2022 · history
Generative AI Mainstreaming
The public release of ChatGPT transformed inference into a ubiquitous, real-time consumer experience, shifting focus toward low-latency model serving.

← all terms