Case transformers

A case transformer converts one or more benchmark-owned source files into the canonical eval-case shape that Trunchbull runs. Transformers are useful when prompts, labels, solutions, or verifier arguments live in different files or use a benchmark-specific schema.

Transformers do not replace evaluators:

raw source files
        ↓
case transformer
        ↓
normalized model prompt + private grading data
        ↓
model trial
        ↓
evaluator

The platform invokes the transformer while inspecting and compiling the pinned repository. Every normalized case is then frozen into the immutable release. At run time, every selected model receives the same compiled prompt, tools, and limits. Private expected answers and custom-evaluator case data are not added to the model prompt.

Repository layout

Keep upstream data separate from authored transformation and evaluation logic:

benchmark/
├── benchmark.config.ts
├── data/
│   ├── cases.jsonl
│   └── labels.jsonl
├── transformers/
│   └── index.ts
└── evals/
    └── benchmark.ts

The eval names its transformer and raw sources. Paths are relative to the eval module:

export default {
  id: "benchmark",
  title: "Benchmark",
  cases: {
    transformer: "../transformers/index.ts",
    sources: {
      cases: "../data/cases.jsonl",
      labels: "../data/labels.jsonl",
    },
  },
  evaluate: { kind: "non_empty" },
}

An eval may declare between one and sixteen named sources. Source formats are not prescribed: the transformer receives their exact text, path, and SHA-256 digest.

Transformer module

transformers/index.ts default-exports a plain object with a name and transform function:

export default {
  name: "Example transformer",

  transform({ sources }) {
    const labels = JSON.parse(sources.labels.content)

    return sources.cases.content
      .split(/\r?\n/)
      .filter(Boolean)
      .map((line) => {
        const row = JSON.parse(line)
        const expected = labels[row.id]

        return {
          id: row.id,
          prompt: row.prompt,
          description: row.description ?? "",
          evaluate: {
            kind: "exact",
            expected,
          },
        }
      })
  },
}

The function may be synchronous or asynchronous. It must return between one and 25,000 ordinary author cases. Each case may define:

• a stable id;
• the model-facing prompt and optional systemPrompt;
• available tools;
• JSON-compatible private data for a module-level custom evaluator;
• a built-in per-case evaluate specification.

Do not place solutions in prompt. Put expected values in the evaluator or in data, which is delivered only to a custom evaluator after model generation.

Validation and determinism

Inspection:

1. loads only the named regular source files;
2. hashes every source;
3. bundles the transformer from the pinned repository;
4. executes it twice with cloned inputs;
5. rejects nondeterministic output;
6. validates every returned author case;
7. rejects duplicate IDs and releases above 25,000 cases;
8. stores the transformer name, paths, source hashes, byte counts, and case count in the immutable release manifest;
9. compiles both the normalized cases and that provenance into the release digest.

Transformer modules use ordinary JavaScript or TypeScript, built-in APIs, and relative imports. They do not import a Trunchbull SDK. Network requests, environment-dependent values, randomness, and current-time behavior should not be used because a release must reproduce exactly from its source commit.

Preview and export locally

Use the same transformer contract before publishing:

pnpm benchmarks:transform -- \
  --transformer=transformers/index.ts \
  --source=cases=data/cases.jsonl \
  --source=labels=data/labels.jsonl \
  --output=exports/cases.jsonl

The command writes normalized JSONL plus exports/cases.jsonl.manifest.json. The manifest records transformer and source hashes, output hash, format, and case count.

Use JSON output when another tool expects an array:

pnpm benchmarks:transform -- \
  --transformer=transformers/index.ts \
  --source=cases=data/cases.jsonl \
  --output=exports/cases.json \
  --format=json

Batch exports

A batch manifest can transform several benchmark ports in one deterministic authoring pass:

{
  "protocolVersion": 1,
  "jobs": [
    {
      "id": "arc-challenge",
      "transformer": "arc-challenge/transformers/index.ts",
      "sources": {
        "cases": "arc-challenge/data/cases.jsonl"
      },
      "output": "arc-challenge/exports/cases.jsonl"
    }
  ]
}

Run every job with:

pnpm benchmarks:transform:batch -- --manifest=transform-batch.json

Paths in a batch manifest are relative to that manifest. The command writes an individual receipt for every export and a combined .report.json.

What belongs elsewhere

A transformer normalizes data; it does not create missing runtime infrastructure.

• Use an evaluator for output grading.
• Use tools/ for callable benchmark resources.
• Mark cases that need a sandbox and provide their environment separately.
• Keep probability-scored tasks unported until the model runtime exposes the required probabilities.
• Use a custom evaluator when deterministic grading cannot be expressed by a built-in evaluator.

The complete ARC-Challenge example in examples/benchmarks/arc-challenge demonstrates raw source data, a transformer, an eval declaration, and an exported normalized fixture.