Eval cases

Independently executed tasks with static or dynamic grading that observe the model through an agent.

An eval case is the smallest independently executed and scored task in a benchmark. It describes what the model receives, what resources the agent can expose, and how the resulting behavior is evaluated.

Every compiled case has:

a stable case ID;
a user prompt and optional system prompt;
zero or more available tool names;
an evaluator;
optional case data for custom evaluation;
source information pointing back to the authoring file and data row.

Case IDs are release-global and stored exactly as authored. They are not automatically prefixed with the eval ID, so duplicate IDs across separate eval modules fail compilation.

The agent is between the case and the model

An eval case does not directly command the model to call a particular function. It configures a trial. The agent exposes the allowed tools, forwards model tool requests to the runtime, returns results to the model, and records the trace.

eval case → agent → model
                ↕
          available tools

final output + tool trace → evaluator → result

The model may call no tools, the correct tool, the wrong tool, or the correct tool with incorrect arguments. Those are behaviors for the evaluator to distinguish.

Static cases

A static case uses a declarative evaluator. Its pass condition is completely described as data and compiled into the release. Static cases are a good fit for exact answers, required phrases, answer banks, and tool-call syntax.

This complete eval asks the model to look up a policy. The tools field makes lookup_policy available; the evaluator—not the tool list—requires the observed call and checks its arguments.

evals/policy-lookup.ts

export default {
  id: "policy-lookup",
  title: "Policy lookup",
  prompt: "Find the return policy for unopened items.",
  tools: ["lookup_policy"],
  evaluate: {
    kind: "tool_call_syntax",
    version: 1,
    alternatives: [
      {
        order: "unordered",
        calls: [
          {
            toolName: "lookup_policy",
            arguments: {
              topic: {
                required: true,
                acceptedValues: ["returns"],
              },
            },
            extraArguments: "forbid",
          },
        ],
        extraCalls: "forbid",
      },
    ],
  },
}

The matching tool can be exported from the benchmark repository:

tools/support-api.ts

import { tool } from "ai"
import { z } from "zod"

export const tools = {
  lookup_policy: tool({
    description: "Look up a customer-support policy by topic.",
    inputSchema: z.object({
      topic: z.string(),
    }),
    execute: async ({ topic }) => ({
      topic,
      policy: "Unopened items may be returned within 30 days.",
    }),
  }),
  get_order: tool({
    description: "Look up an order by ID.",
    inputSchema: z.object({
      orderId: z.string(),
    }),
    execute: async ({ orderId }) => ({
      orderId,
      status: "delivered",
      eligible: true,
    }),
  }),
}

Declare the ordinary packages used by that tool module:

package.json

{
  "name": "support-tool-use-benchmark",
  "private": true,
  "type": "module",
  "dependencies": {
    "ai": "^6.0.0",
    "zod": "^4.0.0"
  }
}

List both entrypoints in the config:

benchmark.config.ts

export default {
  protocolVersion: 1,
  title: "Support tool use",
  version: "1.0.0",
  license: "Apache-2.0",
  tools: ["tools/support-api.ts"],
  evals: ["evals/policy-lookup.ts", "evals/refund-decision.ts"],
}

Other built-in static evaluators include:

{ kind: "non_empty" }
{ kind: "exact", expected: "30 days" }
{ kind: "contains", expected: "30 days" }
{ kind: "answer_bank", answers: ["30 days", "thirty days"] }
{ kind: "gsm8k", expected: "41" } // Upstream-compatible `#### 41` extraction

Dynamic cases

A dynamic case uses a custom evaluator function when the correct result depends on runtime behavior that is awkward to express declaratively. The case itself is still compiled and frozen at publication. What is dynamic is the grading logic applied to the completed trial.

This working example checks both the model's structured answer and its observed tool use:

evals/refund-decision.ts

export default {
  id: "refund-decision",
  title: "Refund decision",
  systemPrompt: "Use account data before making a refund decision.",
  tools: ["get_order"],
  cases: [
    {
      id: "delivered-order",
      prompt:
        "Check order order_123 and return JSON with orderId and eligible.",
      data: {
        expectedOrderId: "order_123",
        expectedEligible: true,
      },
    },
  ],
  evaluate({ output, caseData, toolCalls }) {
    let answer
    try {
      answer = JSON.parse(output)
    } catch {
      return {
        passed: false,
        score: 0,
        message: "The final answer was not valid JSON.",
      }
    }

    const usedExpectedTool = toolCalls.some(
      (call) =>
        call.toolName === "get_order" &&
        call.input?.orderId === caseData.expectedOrderId
    )
    const answerIsCorrect =
      answer.orderId === caseData.expectedOrderId &&
      answer.eligible === caseData.expectedEligible

    return {
      passed: usedExpectedTool && answerIsCorrect,
      score: Number(usedExpectedTool) * 0.5 + Number(answerIsCorrect) * 0.5,
      message:
        usedExpectedTool && answerIsCorrect
          ? null
          : "The answer or supporting tool call was incorrect.",
      details: { usedExpectedTool, answerIsCorrect },
    }
  },
}

The evaluator receives the final output, JSON-compatible caseData, raw toolCalls, normalizedToolCalls, finishReason, and usage data. It returns a boolean or an object containing passed, an optional score from 0 to 1, and optional evidence.

Custom evaluators run after generation in a pinned, private Worker. Like custom tools, they have no outbound internet access. They also have a five-second deadline and no platform secrets. A crash, timeout, or invalid result is recorded as evaluator infrastructure failure rather than model failure.

Dynamic case generation is different

Trunchbull currently supports dynamic grading, not per-run generation of new case inputs. Inline, CSV, and JSONL cases are expanded and frozen when the benchmark is published. This ensures every compared model receives the same tasks.

One eval can expand into many cases

An eval module can hold one prompt, define several inline cases, or load a CSV or JSONL data file. Shared fields are inherited by its cases:

evals/refunds.ts

export default {
  id: "refunds",
  systemPrompt: "Answer with the policy duration.",
  tools: ["lookup_policy"],
  evaluate: { kind: "contains" },
  cases: "./refunds.csv",
}

evals/refunds.csv

id,prompt,expected
unopened,"What is the unopened-item return window?","30 days"
holiday,"What is the holiday return window?","60 days"

Each row becomes an independent case and receives its own trial and result. CSV supports only id, prompt, expected, and description, and only with an eval-level non_empty, exact, or contains evaluator. Inline and JSONL cases can additionally override tools, system prompts, private data, and declarative evaluators.

Choosing static or dynamic grading

Prefer a static evaluator when the requirement can be expressed declaratively. It is easier to inspect, audit, and reproduce. Use a dynamic evaluator when you need to combine evidence, parse structured output, apply domain logic, or grade sequences of agent actions.

Next, learn how case transformers normalize benchmark-specific sources before those cases become an immutable release in publishing and credit. See the authoring components reference for the complete field, inheritance, path, and limit contracts.

The agent is between the case and the model

Static cases

Dynamic cases

One eval can expand into many cases

Choosing static or dynamic grading

On this page