Gradient descent
A method for training AI by making tiny, repeated adjustments to its internal settings, always moving in the direction that reduces its mistakes until it gets the best possible result.
An iterative optimization algorithm used to train neural networks by calculating the gradient of the loss function with respect to model parameters and updating those parameters in the opposite direction to minimize error.
A first-order iterative optimization algorithm for finding the local minimum of a differentiable objective function; it updates parameters via the rule θ = θ - η∇J(θ), where η is the learning rate and ∇J(θ) is the gradient of the loss function, effectively navigating the loss landscape toward lower error.
evolution
- 1847 · historyMethod of Steepest Descent
Augustin-Louis Cauchy published the first formal description of the steepest descent method for finding local minima.
- 1944 · historyCurry's Optimization
Haskell Curry provided the first convergence analysis for the method of steepest descent in Hilbert spaces.
- 1986 · historyBackpropagation Popularization
Rumelhart, Hinton, and Williams demonstrated that gradient descent could efficiently train multi-layer neural networks.
- 2011 · historyNesterov Accelerated Gradient
Yurii Nesterov's momentum-based optimization techniques became standard for accelerating convergence in deep learning.
- 2014 · historyAdam Optimizer
Kingma and Ba introduced Adam, an adaptive moment estimation algorithm that became the default optimizer for most modern AI models.