Technique
Prefix Tuning
Learning trainable key/value prefixes prepended to every transformer layer.
Definition
Prefix tuning (Li & Liang 2021) extends prompt tuning by learning soft prefixes injected into the keys and values of every attention layer, not just the input. It rivals full fine-tuning at <1% of parameters and is the conceptual ancestor of P-Tuning v2.
Common use cases
- Multi-task serving
- Cheap adaptation
- Research