Authoring components

Trunchbull authoring repositories use ordinary JavaScript or TypeScript. They do not import a Trunchbull SDK. The platform discovers or reads the entrypoints, bundles their local imports, validates their exports, expands every case, and freezes the result into an immutable release.

Repository layout and discovery

The smallest repository needs only an eval module:

benchmark/
└── evals/
    └── answer.ts

benchmark.config.ts is optional. With no explicit evals or tools arrays, Trunchbull recursively discovers .ts, .mts, .js, and .mjs modules under evals/ and tools/, excluding files whose names contain .test. or .spec.. The directories are resolved beside the config file, which allows a benchmark to live inside a monorepo.

Use explicit entrypoint arrays for a stable, reviewable release boundary:

export default {
  protocolVersion: 1,
  title: "Support quality",
  version: "1.0.0",
  evals: ["evals/refunds.ts"],
  tools: ["tools/support.ts"],
}

All configured paths use repository-relative POSIX syntax and are relative to the directory containing benchmark.config.ts. An explicit evals array must contain at least one module. An omitted tools array is valid for benchmarks that do not expose tools.

Config

The config default-exports one plain, JSON-compatible object. It must not run setup code or contain functions.

Unknown fields are rejected. Authoring IDs must start with a lowercase letter or number and may then use lowercase letters, numbers, dots, underscores, and hyphens. They are limited to 120 characters.

Eval modules

Every eval module default-exports one object:

export default {
  id: "refunds",
  title: "Refund policy",
  description: "Checks policy accuracy.",
  systemPrompt: "Answer briefly.",
  tools: ["lookup_policy"],
  evaluate: { kind: "contains", expected: "30 days" },
  cases: [
    {
      id: "refunds.standard",
      prompt: "What is the standard return window?",
    },
  ],
}

An eval requires id. It may define title, description, systemPrompt, tools, and an evaluator shared by its cases. It then uses exactly one of these shapes:

• prompt plus evaluate for one case whose case ID equals the eval ID;
• cases: [...] for inline cases;
• cases: "./cases.csv" or cases: "./cases.jsonl" for a sibling data file;
• cases: { transformer, sources } for authored transformation.

Eval modules may use local imports, but their inspected value must otherwise be plain JSON-compatible data. The only supported function in that export is one module-level custom evaluate function. Per-case evaluator functions are not supported.

Cases and inheritance

Every case requires a release-global id and a non-empty prompt. Case IDs are stored exactly as authored; Trunchbull does not prefix them with the eval ID. A duplicate ID in any two eval modules fails compilation. Prefix IDs yourself when several evals use generic names such as valid or default.

Cases may also set description, systemPrompt, tools, JSON-compatible private data, and a declarative evaluate specification. Inheritance is field-specific:

The compiled title is the eval title or ID for a one-case eval. For expanded cases it is <eval title or ID> — <case ID>, truncated to 120 characters. Case data is private grading input and is not appended to the model prompt. A release may contain at most 25,000 compiled cases.

CSV cases

CSV supports only the columns id, prompt, expected, and description. id and prompt are required. The eval-level evaluator must be non_empty, exact, or contains; exact and contains also require expected on every row. Use JSONL when cases need tools, system prompts, private data, or case-specific evaluator kinds.

JSONL cases

Each non-empty line is one complete case object and can use every case field. The file must contain at least one case. Data-file paths are relative to the eval module, not the config.

Tools

A tool module exports a named tools object. Each value must be an executable AI SDK tool with a non-empty description, an input schema, and an execute function:

import { tool } from "ai"
import { z } from "zod"

export const tools = {
  lookup_policy: tool({
    description: "Look up a policy by topic.",
    inputSchema: z.object({ topic: z.string() }),
    execute: async ({ topic }) => ({ topic, windowDays: 30 }),
  }),
}

Declare ordinary dependencies in package.json; Trunchbull installs from the repository lockfile when one exists. Tool names must be unique across the release. Every name requested by a case must exist in the inspected tool set, or compilation fails.

Evaluators

Built-in evaluator shapes are:

{ kind: "non_empty" }
{ kind: "exact", expected: "30 days" }
{ kind: "contains", expected: "30 days" }
{ kind: "answer_bank", answers: ["30 days"], forbiddenAnswers: ["60 days"] }
{ kind: "gsm8k", expected: "41" }
{ kind: "tool_call_syntax", version: 1, alternatives: [/* plans */] }

exact compares trimmed output exactly. contains performs a case-insensitive substring check after trimming the expected value. answer_bank performs normalized, case-insensitive substring matching and fails first when a forbidden answer appears. gsm8k requires a normalized numeric expected value.

A custom evaluator is one module-level evaluate function on the eval object. It receives output, caseData, raw toolCalls, normalizedToolCalls, finishReason, and usage. It may return a boolean or an object with passed, optional score from 0 to 1, optional message, and optional JSON-compatible details. Custom evaluators have a five-second execution deadline; invalid results, crashes, and timeouts are infrastructure failures.

Transformers

A transformer is selected by an eval's cases object. Its path and every source path are relative to that eval module. It default-exports { name, transform }; transform receives named sources containing exact content, path, and sha256, and returns ordinary case objects.

Each transformer accepts 1–16 named sources. An individual source is limited to 50 MB, all transformer sources in the repository are limited to 200 MB, and the transformed release remains subject to the 25,000-case limit. Trunchbull runs the transformation twice and rejects nondeterministic output.

For worked examples, continue with eval cases and case transformers.