Loss function
Think of this as a 'mistake meter' for the computer. It measures how far off the model's guess is from the right answer, and the goal of training is to keep turning the dial down until the mistakes are as small as possible.
A mathematical function that quantifies the error between a model's predicted output and the actual ground truth. During the training process, optimization algorithms like gradient descent iteratively adjust the model's parameters to minimize this value.
A scalar-valued objective function, denoted as L(y, ŷ), that maps the discrepancy between predicted outputs and target labels to a non-negative real number. Training constitutes a constrained or unconstrained optimization problem where the model parameters are updated via backpropagation to minimize the expected risk over the training distribution.
evolution
- 1805 · historyMethod of Least Squares
Adrien-Marie Legendre published the first formal method for minimizing the sum of squared residuals, the foundation for modern loss functions.
- 1958 · historyPerceptron Learning Rule
Frank Rosenblatt introduced a simple error-correction rule to adjust weights based on the difference between predicted and actual output.
- 1986 · historyBackpropagation Popularization
Rumelhart, Hinton, and Williams demonstrated how to use gradient descent to minimize error across multi-layer neural networks.
- 2012 · historyCross-Entropy in Deep Learning
The success of AlexNet popularized the use of categorical cross-entropy as the standard loss function for large-scale image classification.