Metric & Benchmark
Synthetic Benchmark
Evaluation set generated by an LLM rather than collected from humans.
Definition
Synthetic benchmarks scale evaluation cheaply but risk biasing toward whichever model generated them. They are useful for capability triage and contamination-resistant testing when paired with human-authored reference benchmarks.
Common use cases
- Triage eval
- Capability probing
- Contamination tests