Concept
aka attention mechanism
Attention
Mechanism letting a model weight different parts of its input when producing each output token.
Definition
Attention is the core operation in Transformer models. For every output position, the model computes a weighted sum over input positions, where the weights (attention scores) measure how relevant each input is. Self-attention compares every token to every other token in the same sequence, enabling long-range dependencies that recurrent networks struggled with.
Common use cases
- Language modeling
- Machine translation
- Vision transformers