Architecture
aka Attention with Linear Biases
ALiBi
Position-encoding method that adds a fixed linear distance penalty to attention scores.
Definition
ALiBi (Press et al. 2022) replaces position embeddings with a per-head linear bias proportional to query-key distance. It extrapolates to much longer sequences than seen during training and is used in MPT and BLOOM-Z.
Common use cases
- Long-context
- Out-of-distribution length