Glossary

Benchmark

The top-level package users author, publish, version, share, and select for a run. It contains one or more eval cases plus their tools, evaluators, defaults, metadata, and provenance.

Benchmark release

An immutable compiled version of a benchmark. A release is identified by its package ID and version and pinned to a repository commit and content digest.

Eval

An authoring module under evals/. One eval can describe a single case or expand into many cases from inline data, CSV, or JSONL.

Eval case

The smallest independently executed and scored task. A case includes a prompt, available tool names, an evaluator, and optional case data.

Static case

An eval case graded by a built-in declarative evaluator, such as exact match, contains, answer bank, or tool-call syntax.

Dynamic case

An eval case graded by a custom evaluator function after the trial completes. In the current authoring contract, “dynamic” refers to grading logic, not per-run generation of new prompts.

Agent

The execution loop between the model and its environment. It presents prompts and tool schemas, executes valid model-requested tool calls, returns tool results, and records the trace.

Model

The system being evaluated. The model chooses its response and whether to request any tools exposed by the agent.

Tool

A model-callable capability with a name, description, input schema, and implementation. A case can make a tool available; the model chooses whether and how to call it.

Evaluator

The logic that turns a completed trial into pass/fail, an optional normalized score, and supporting evidence. Evaluators may inspect final output, tool behavior, or both.

Run

One execution plan applying selected models and run configuration to a benchmark release.

Trial

One model executing one eval case once. Repetitions create multiple trials for the same model and case.

Result

The recorded outcome of a trial, including evaluator evidence, output, tool trace, usage, cost, latency, and failure classification.

Infrastructure failure

A trial that could not produce a valid model result because the provider, agent, tool runtime, sandbox, or evaluator failed. It is reported separately from behavior the model attempted and got wrong.

Package ID

The stable repository-derived identity of a published benchmark, such as acme/support-quality.

Source digest

A cryptographic hash of the canonical compiled release. It proves that the published content matches the content reviewed during inspection.

Publisher

The signed-in Trunchbull account that publishes a repository release. The publisher owns the Trunchbull release record; source attribution remains tied to the GitHub repository and commit.

Provenance

The information connecting a release to its source: repository URL, GitHub owner and repository, commit SHA, config path, package version, digest, and license.