Publishing and credit

Publishing turns an authoring repository into an immutable benchmark release that Trunchbull can run repeatedly. It also establishes the release's identity, source provenance, ownership, and license.

What you publish

Trunchbull publishes a benchmark package, not a loose collection of prompts. The package contains its expanded eval cases, compiled tool contracts, custom evaluators, defaults, metadata, and source identity.

A publishable repository can include:

benchmark/
├── benchmark.config.ts
├── data/
├── evals/
├── transformers/
├── tools/
├── README.md
├── LICENSE
└── package.json

The config supplies release metadata and can define explicit entrypoints:

export default {
  protocolVersion: 1,
  title: "Support quality",
  version: "1.0.0",
  description: "Support correctness and tool-use checks.",
  license: "Apache-2.0",
  homepage: "https://github.com/acme/support-quality",
  tags: ["support", "tool-use"],
  tools: ["tools/index.ts"],
  evals: ["evals/refunds.ts"],
}

The config itself is optional. When evals or tools is omitted, Trunchbull discovers supported JavaScript and TypeScript modules beneath the corresponding directory beside the config. Explicit arrays are recommended when the release boundary should not change as helper modules are added.

Inspection comes before publication

The publishing screen accepts a public GitHub repository plus an optional branch, tag, commit, and config path.

Inspection:

1. resolves the requested source to a full commit SHA;
2. discovers or reads the configured tool and eval entrypoints;
3. installs ordinary repository dependencies in an isolated builder;
4. validates exports and expands inline, CSV, and JSONL cases;
5. runs declared case transformers against their named pinned sources;
6. validates and freezes every normalized case;
7. compiles tools and custom evaluators;
8. produces the complete release manifest and content digest.

Inspection is read-only. It creates no release and deploys nothing. Review the resolved commit, metadata, cases, tools, evaluators, license, and digest before publishing.

Publication freezes the reviewed result

When you publish, Trunchbull rebuilds the same commit. The compiled digest must match the inspected digest. If it does not, publication stops and asks you to inspect again.

After the digest matches, Trunchbull:

1. records the repository source and signed-in publishing account;
2. creates the immutable package version;
3. stores every compiled case;
4. queues pinned deployments for tools and custom evaluators;
5. materializes the runnable cases after every dependency is ready.

A release progresses through pending, compiling, and deploying_tools before becoming ready. A failed tool or evaluator deployment keeps the release unavailable rather than publishing a partially runnable benchmark.

Who gets credit

Credit has several parts, and they should not be collapsed into one name:

The package ID and source link make the originating repository visible even when another team runs the benchmark. The exact commit makes the credited source auditable rather than merely naming an organization.

If several people contributed, keep the repository contributor history and add an AUTHORS.md or equivalent credit section to the README. The current benchmark manifest does not have a structured contributors field, so repository documentation is the place to record individual contributors.

Forks and adaptations

A fork is a different GitHub repository and therefore receives a different package ID. Its publisher should:

• explain what changed in the README;
• retain upstream copyright and license notices;
• link to the upstream benchmark;
• use a version that describes the fork's own release history;
• avoid implying that upstream authors endorsed the modified benchmark.

Trunchbull preserves the fork's direct source provenance. Because the current manifest has no structured derivedFrom field, the README and homepage should carry upstream attribution explicitly.

Immutability and retries

The same package version cannot refer to different compiled content. Publishing the same source, commit, config path, version, and digest is idempotent. A failed pinned deployment may be retried without creating a different logical release.

Changes require a new commit and normally a new semantic version. Historical runs remain attached to the exact release they used.

For the full repository contract and step-by-step workflow, continue with Create and publish your first benchmark, or use the authoring components reference for the exact export shapes, inheritance rules, paths, and limits.