Benchmarks
The single publishable unit that packages eval cases, tools, grading, metadata, and reproducibility into one release.
A benchmark is the unit you author, publish, version, share, and run. It packages one or more eval cases together with the resources and metadata needed to execute them consistently.
A benchmark is not one prompt and it is not one model response. Think of it as a small, reproducible test suite for model behavior:
Benchmark
├── identity and version
├── eval cases
├── tool definitions, when needed
├── evaluators
├── execution defaults
└── source, license, and attributionEach eval case becomes an independently executed and scored task. A run applies the same published benchmark release to every selected model, producing one or more trials per case and model.
What belongs in a benchmark
| Part | Purpose |
|---|---|
| Metadata | Gives the benchmark a package ID, title, version, description, tags, license, and homepage. |
| Eval cases | Describe the tasks and the behavior to grade. |
| Tools | Define capabilities the agent can expose to the model during a trial. |
| Evaluators | Turn the model's output and behavior into a result with evidence. |
| Defaults | Set shared system instructions and execution ceilings. |
| Source identity | Pins the repository, commit, config path, and compiled digest. |
The repository is the authoring format. The published release is the runnable product.
From benchmark to result
The execution flow matters, especially for tool-use benchmarks:
- Trunchbull loads an immutable benchmark release.
- The eval case supplies the task, case data, evaluator, and names of tools that may be exposed.
- The agent presents the task and available tool schemas to the model.
- The model chooses whether to answer, call a tool, or take several steps.
- The agent executes valid tool calls and records the complete trace.
- The evaluator grades the final output, the recorded behavior, or both.
- Trunchbull stores the result, evidence, usage, cost, latency, and any infrastructure failure separately.
Cases do not choose tools for the model
A case can define the set of tools available during its trial. The model, through the agent loop, chooses whether and how to use them. The evaluator decides whether that behavior was correct.
Authoring source and compiled release
A benchmark repository uses ordinary JavaScript or TypeScript. Authors do not install a Trunchbull SDK or CLI. A small repository can look like this:
support-quality/
├── benchmark.config.ts
├── evals/
│ ├── policy-answer.ts
│ └── refund-decision.ts
└── tools/
└── support-api.tsDuring inspection, Trunchbull resolves the repository to a full commit, validates its exports, expands its cases, bundles tools and custom evaluators, and calculates a digest. Publishing repeats that build and rejects it if the result differs from what was inspected.
The resulting release is immutable. Runs use its stored cases and pinned deployments; they do not fetch mutable files from the repository.
Versions are part of the meaning
Changing a prompt, expected answer, tool schema, evaluator, or limit can change the result. That is a new benchmark release, not a silent edit to an old one.
Use semantic versions when you declare a version:
- patch for corrections that should not materially change the measured skill;
- minor for additive cases or compatible coverage;
- major for changes that alter the benchmark's meaning or comparability.
Trunchbull prevents the same package version from being published with different compiled content.
What a benchmark does not contain
A benchmark does not select the model under test. Models, repetitions, and run-wide budgets belong to the run configuration. This separation lets one benchmark release be applied fairly to many models.
Continue with eval cases, then learn how publishing and credit work.