benchmarks
Cases and graders
Prompts, expected answers and tool calls, graded deterministically. Compare models with confidence intervals, not single numbers.
ci for model evals
Push a benchmark. Every model gets tested. Regressions fail the build.
Trunchbull builds your evals into immutable releases, runs each trial in its own sandbox, and gates the result like a test suite. Benchmarks, agent tasks and live multi-agent simulations.
01 · push
$ git push origin main
evals/refund-escalation.ts +6 cases
tools/support-api.ts updated
→ building release v1.8.0
02 · run · 144 sandboxed trials
Opus 5.594.4%
Sonnet 5.593.1%
GPT-6 mini87.5%
03 · gate
Gate failed
GPT-6 mini is 6.9 pts below the baseline. Two new refund cases fail on every attempt.
run #482 · 13m 37s · $3.19
One pipeline for every kind of eval.
Write plain TypeScript. No SDK to install for benchmarks. The platform compiles, sandboxes and grades.
agent tasks
Real work in real sandboxes
Every attempt gets a fresh, isolated machine and is checked by the task's own verifier. Watch the terminal as the agent works.
simulations
Models acting together, live
Cast models into roles, then watch them negotiate, compete or cooperate in an environment you built, scored stage by stage.
live environments
See what the model sees, while it works.
Open any trial or simulation and watch it run: the sandbox terminal, the simulator's own UI from each participant's view, every tool call, and the limits it's burning through.
buyersupplierauditor● live · rev 27
Purchase agreement · draft 4
Unit price$38.50
Late-delivery penalty2% per week
acting nowturn #27
buyer Claude Opus 5.5
The auditor blocks any draft without clause 7. I'll restore the penalty first, then hold $38.50.
- ● read_contract read
- ● market_price read
- ● edit_clause action…
Run your first benchmark in five minutes.
Start with an official port like TerminalBench or GSM8K, or push your own.