Metric & Benchmark
HumanEval
164-problem coding benchmark from OpenAI testing function generation from docstrings.
Definition
HumanEval is a hand-written code-generation benchmark of 164 Python problems with unit tests. pass@1 measures whether the model's first sample passes all tests. Modern coding LLMs score 80–95%, so harder extensions like SWE-Bench and LiveCodeBench have taken over.
Common use cases
- Code benchmarks
- Model selection for coding