Training
Training is like teaching a student by showing them many examples and letting them practice until they get the right answers more often.
Training is the iterative process of updating a model's internal parameters based on a dataset to reduce error and improve performance on specific tasks.
The process of adjusting a model's weights to minimise a loss on data: pre-training builds general capability, while post-training (fine-tuning, RLHF) shapes behaviour and alignment.
evolution
- 1958 · historyPerceptron Learning Rule
Frank Rosenblatt introduced the first algorithm for training a single-layer neural network using iterative weight adjustments.
- 1986 · historyBackpropagation Popularization
Rumelhart, Hinton, and Williams demonstrated that backpropagation could efficiently train multi-layer neural networks.
- 2006 · historyDeep Belief Networks
Geoffrey Hinton introduced greedy layer-wise pre-training, enabling the effective training of deeper neural architectures.
- 2017 · historyTransformer Architecture
The introduction of the Transformer model enabled massive parallelization of training, shifting the paradigm toward large-scale pre-training.
- 2020 · historyFew-Shot Learning via Scaling
GPT-3 demonstrated that training massive models on diverse datasets allows for emergent capabilities without task-specific fine-tuning.