Overfitting
Overfitting is like a student who memorizes the exact answers to a practice test instead of learning the actual subject; they get a perfect score on the practice test but fail when they see new questions on the real exam.
Overfitting occurs when a model learns the specific noise and random fluctuations in the training data rather than the underlying signal, resulting in high accuracy on training samples but a significant drop in performance when evaluated on unseen validation or test data.
Overfitting is a failure mode in statistical learning where a model captures stochastic noise and idiosyncratic features of the training distribution rather than the underlying data-generating process, characterized by a low empirical risk on the training set coupled with poor generalization performance on the out-of-sample data distribution.
evolution
- 1950 · historyEarly Statistical Recognition
The concept of 'curve fitting' and the dangers of excessive parameters were identified in early statistical modeling literature.
- 1989 · historyFormalization in Neural Networks
Researchers like Geman et al. formally defined the bias-variance tradeoff, providing a mathematical framework for understanding overfitting.
- 1995 · historyIntroduction of Regularization
Techniques like Weight Decay and early stopping became standard practices to penalize model complexity and prevent memorization.
- 2012 · historyDropout Regularization
The introduction of Dropout significantly improved the training of deep neural networks by randomly disabling neurons to prevent co-adaptation.
- 2017 · historyDouble Descent Phenomenon
Research identified that modern over-parameterized models can exhibit a 'double descent' curve, challenging traditional views on overfitting.