Backpropagation
A process that helps an AI learn by looking at its mistakes and working backward through its internal connections to figure out exactly how much each one contributed to the error, so it can make small adjustments to get better next time.
The fundamental algorithm used to train neural networks by calculating the gradient of the loss function with respect to each weight, allowing the model to update its parameters via gradient descent to minimize prediction error.
An efficient application of the chain rule of calculus to compute the partial derivatives of the objective function with respect to every weight in a neural network, enabling the backward propagation of error signals to facilitate gradient-based optimization.
evolution
- 1970 · historySeppo Linnainmaa's Thesis
Seppo Linnainmaa published the first general method for automatic differentiation of discrete connected networks of nested differentiable functions.
- 1986 · historyRumelhart, Hinton, and Williams
The seminal paper 'Learning representations by back-propagating errors' popularized the algorithm by demonstrating its effectiveness in training multi-layer neural networks.
- 1989 · historyLeCun's Handwritten Digit Recognition
Yann LeCun successfully applied backpropagation to convolutional neural networks to recognize handwritten ZIP codes, proving its practical utility.
- 2012 · historyAlexNet and Deep Learning
The success of AlexNet on the ImageNet challenge solidified backpropagation as the standard engine for training deep learning architectures on GPUs.