Concept
Softmax
Function that normalises a vector of logits into a probability distribution summing to one.
Definition
Softmax exponentiates each logit and divides by the sum, producing a valid probability distribution over the vocabulary. Temperature divides the logits before softmax, sharpening or flattening the distribution. It sits between every transformer's final layer and the sampler.
Common use cases
- Final layer
- Classification head
- Attention weights