Architecture
Retrieval-Augmented Transformer
Transformer with a retrieval module integrated into its layers, not just its prompt.
Definition
RETRO and successor architectures bake retrieval into the model: cross-attention layers consume embeddings of retrieved chunks. This contrasts with prompt-level RAG and can be more parameter-efficient for knowledge-heavy tasks.
Common use cases
- Knowledge-intensive QA
- Long-tail facts
- Domain expertise