Attention mechanism
A way for an AI to read a sentence by deciding which words are most important to look at together, like how you might focus on specific clues to understand the meaning of a complex paragraph.
A mechanism that calculates dynamic relevance scores between tokens in a sequence, allowing the model to weigh the importance of different inputs relative to one another when generating an output.
The operation that computes context-dependent weights between elements of a sequence, letting a model focus on the most relevant tokens; self-attention is the core of the Transformer.
evolution
- 2014-09 · historyBahdanau et al. introduction
The attention mechanism was first introduced to allow neural machine translation models to focus on relevant parts of the input sequence.
- 2015-02 · historyLuong et al. refinement
Introduced global and local attention mechanisms, standardizing the dot-product and general scoring functions.
- 2017-06 · historyAttention Is All You Need
The Transformer architecture replaced recurrent layers with multi-head self-attention, enabling parallel processing and long-range dependency modeling.
- 2020-10 · historyVision Transformer (ViT)
Demonstrated that pure attention mechanisms could be applied to image patches, effectively replacing convolutional neural networks in computer vision.