Architecture
aka Rotary Position Embedding
RoPE
Position-encoding scheme that rotates query/key vectors in 2D subspaces by their position.
Definition
Rotary position embedding (Su et al. 2021) encodes positions as rotation matrices applied to query/key pairs, naturally generalising to longer contexts than seen at training. RoPE is used in Llama, Qwen, Mistral and most modern decoder-only LLMs.
Common use cases
- Long-context extension
- Modern LLMs
- Position embeddings