Run configuration

A high-quality run configuration does two things: it gives every model the same workload and gives that workload enough budget to finish. Configure the run only after the benchmark cases, evaluators, tools, and execution lane are correct.

Start with the workload

Trunchbull creates one trial for every selected case, model, and repeat:

trial count = selected cases × selected models × repeats

For example:

12 cases × 3 models × 2 repeats = 72 trials

This number drives token use, cost, tool-call capacity, and run duration. Confirm the trial count in the run summary before increasing any limit.

Select comparable models

The run builder supports up to five selected models. Every selected model receives the same frozen cases, prompts, tools, and run limits.

Choose one selected model as the baseline. A useful baseline is the model currently in production or the last candidate your team approved. The baseline does not receive special execution behavior; it is the reference used to read candidate deltas.

Choose the repeat count

Repeats per Case & Model runs each case-model pair independently from 1 to 10 times. The builder shows the resulting total before any limit is configured.

Do not use repeats to compensate for weak cases. First make each case specific, gradeable, and representative.

Set execution limits

The builder groups limits by scope: Per Model Call, Per Trial, and Whole Run. Treating every field as “per trial” can make a run stop much earlier than expected.

Model steps

Set Max Model Steps to the longest legitimate path through one case, plus a small allowance for recovery. A direct-answer case usually needs one step. A tool-use case commonly needs one tool request and one final answer. Large values do not improve a task that should finish quickly; they allow unproductive loops to run longer.

Tool calls

Max Tool Calls is shared by the whole matrix. If a benchmark requires one tool call in every trial, the minimum for the 72-trial example is:

72 trials × 1 required call = 72 minimum tool calls

Add capacity only for legitimate optional calls or retries. Trunchbull prevents launch when the ceiling is below the minimum implied by required-tool checks.

Output tokens

Max Output Tokens limits a single generated response, not the complete run. Size it to the expected answer:

classification or exact answer:  32–128
short grounded response:         256–512
structured or explanatory task:  512–2,048

A larger ceiling is appropriate only when the evaluator expects longer output. It should not be used as a substitute for a focused prompt.

Total tokens

Max Total Tokens is the shared budget for the full matrix. The run builder calculates a recommendation from the execution lane, trial count, model steps, output ceiling, and attached tool schemas.

Use the recommendation for the first representative run. If you lower it, verify that cases still finish; a low shared ceiling can make later trials incomplete even when every per-call output limit is reasonable.

Spend

Max Run Spend is the hard run-wide cost stop. Estimate it from a small representative run, then include headroom for provider price differences and tool-follow-up steps.

Token and spend ceilings are independent. The run stops when either budget is exhausted, so both must be large enough for the intended workload.

Configuration examples

Small direct-answer comparison

Cases:                       20
Models:                       2
Repeats:                      2
Total trials:                80
Max Model Steps:              1
Max Tool Calls:               0
Max Output Tokens:          128
Max Total Tokens:           use the builder recommendation

Tool-use regression suite

Cases:                       12
Models:                       3
Repeats:                      2
Total trials:                72
Required tool calls/trial:    1
Max Model Steps:              3
Max Tool Calls:              90
Max Output Tokens:          512
Max Total Tokens:           use the builder recommendation

The extra 18 tool calls allow valid multi-call behavior without turning the tool budget into an unbounded retry pool.

What the run configuration does not change

The run builder does not redefine benchmark quality. These remain part of the selected benchmark release:

• prompts and case data;
• built-in or custom evaluators;
• required tools and tool schemas;
• default execution lane;
• benchmark-level execution defaults.

If a grader, prompt, or case is wrong, publish a corrected benchmark version. Do not tune run budgets until an invalid benchmark happens to pass.

Frozen run contract

When the run launches, Trunchbull stores an immutable configuration snapshot and hash. The snapshot binds the selected models, cases, benchmark versions, tools, lane, repeats, and budgets to the resulting transcripts and scores.

Use the run's Configuration tab when reviewing or comparing results. Two runs are directly comparable only when their frozen contracts differ in ways the experiment intended.

□ Selected cases represent the behavior under test
□ Every model is compatible with the benchmark lane and tools
□ The production or approved model is selected as baseline
□ Trial count matches cases × models × repeats
□ Repeat count is intentional
□ Total trials match cases × models × repeats
□ Step ceiling fits the longest valid case path
□ Run-wide tool budget covers required calls across all trials
□ Output ceiling fits one expected response
□ Total token and spend budgets cover the complete run
□ The run summary changes only the intended experimental variable