Architecture
Transformer
Neural architecture built on stacked self-attention layers, the basis of all modern LLMs.
Definition
Introduced in the 2017 Attention Is All You Need paper, the Transformer replaced recurrence and convolution with stacked self-attention and feed-forward blocks. Its parallelism made web-scale training tractable; nearly every frontier text, image, and code model is a Transformer variant.
Common use cases
- Language modeling
- Vision
- Speech