Transformer
A Transformer is a type of AI brain that reads an entire sentence at once rather than word-by-word. It uses a special 'attention' mechanism to figure out which words in a sentence are most important to each other, allowing it to understand context much faster and more accurately than older systems.
A neural network architecture that replaces recurrent structures with a self-attention mechanism. By processing input sequences in parallel rather than sequentially, it calculates the relevance of every token to every other token in the sequence, enabling efficient training and superior contextual modeling for large-scale language tasks.
A neural-network architecture (Vaswani et al., 2017) built on self-attention that processes all tokens in parallel and weighs their pairwise relevance; the foundation of modern LLMs and most generative models.
evolution
- 2017-06 · historyAttention Is All You Need
Google researchers introduced the Transformer architecture, replacing recurrent neural networks with a self-attention mechanism.
- 2018-10 · historyBERT
Google released BERT, demonstrating the power of bidirectional Transformer encoders for state-of-the-art language understanding.
- 2019-02 · historyGPT-2
OpenAI showcased the scalability of Transformer decoders, proving that large-scale generative pre-training leads to emergent capabilities.
- 2020-05 · historyGPT-3
The release of GPT-3 established the paradigm of few-shot learning and massive-scale parameter scaling in Transformer models.
- 2021-10 · historyVision Transformer (ViT)
The architecture was successfully adapted for computer vision, proving Transformers could outperform traditional convolutional networks on image tasks.