Architecture
aka MHA
Multi-head Attention
Attention applied in parallel across several projections, each capturing different relations.
Definition
Multi-head attention splits the model dimension into multiple smaller heads, each computing its own scaled dot-product attention. The results are concatenated and projected, letting the network attend to different relational patterns simultaneously — syntactic, semantic, positional.
Common use cases
- Every transformer
- Vision models
- Speech models