Architecture
aka RNN
Recurrent Neural Network
Sequence model that processes inputs one step at a time, carrying a hidden state.
Definition
RNNs (LSTM, GRU) were the dominant sequence model before transformers. They process tokens sequentially, accumulating a hidden vector. While elegant, they struggle to model very long dependencies and cannot be parallelised across time, which is why transformers displaced them in NLP.
Common use cases
- Edge sequence tasks
- Speech
- Legacy NLP