Recurrent neural network
A type of AI designed to understand information that comes in a specific order, like words in a sentence, by reading one piece at a time and remembering what it just saw to help it understand the next part. While it was once the standard for things like translation, it has mostly been replaced by newer, faster technology.
A neural network architecture that processes sequential data by iterating through time steps, using a hidden state to pass information from one step to the next. While effective for capturing temporal dependencies, it is now largely superseded by Transformer models due to their superior parallelization capabilities and ability to handle long-range dependencies.
A class of artificial neural networks characterized by recurrent connections that form directed cycles, enabling the maintenance of an internal hidden state that functions as a memory of past inputs. By processing sequences step-by-step, the architecture models temporal dynamics; however, it is constrained by sequential computation dependencies and vanishing gradient issues, leading to its widespread displacement by Transformer architectures utilizing self-attention mechanisms.
evolution
- 1982 · historyHopfield Network
John Hopfield introduced recurrent networks as associative memory systems, establishing the foundational concept of feedback loops.
- 1986 · historyBackpropagation Through Time (BPTT)
Rumelhart, Hinton, and Williams popularized the algorithm allowing recurrent networks to be trained via gradient descent.
- 1997 · historyLong Short-Term Memory (LSTM)
Hochreiter and Schmidhuber introduced the LSTM architecture to solve the vanishing gradient problem, enabling learning of long-range dependencies.
- 2014 · historySequence-to-Sequence (Seq2Seq)
Sutskever et al. demonstrated the power of encoder-decoder RNN architectures for machine translation, marking the peak of RNN dominance.