Skip to Main Content
Trunchbull
Sign inSign up
PUBLIC REGISTRY
Trunchbull portNo sandbox required

ARC-Challenge

Upstream publisher Allen Institute for AIallenai/ARC

A clear, no-frills test of science reasoning on challenging grade-school questions with multiple-choice answers.

sciencereasoningmultiple-choice
PROVENANCEIMMUTABLE RELEASE
Upstream source
allenai/ARC
Trunchbull port
TrunchbullCI/examples
Version
ARC / TRAIN
Release digest
sha256:8610…e223
License
CC BY-SA 4.0
Source commit: dataset-2026-07Published July 28, 2026

WHAT THIS PROJECT MEASURES

README

ARC-Challenge is the difficult partition of the AI2 Reasoning Challenge. Each case presents a science question and a fixed set of answer choices.

Trunchbull exposes a small deterministic demo port. The grader extracts a final answer choice and compares it with the published answer key.

IN PLAIN ENGLISH

ARC-Challenge is built for questions that are harder than simple recall. Each case gives the model a science question and a fixed set of choices; the challenge is choosing the answer that follows from the underlying concept. Trunchbull CI packages that contract with pinned provenance and transparent grading.

EXECUTION CONTRACT

Runs in the standard chat lane with exact answer-choice grading.

QUICK ANSWERS

ARC-Challenge, explained

What does ARC-Challenge measure?

ARC-Challenge measures science knowledge and reasoning on difficult grade-school questions designed to resist shallow retrieval and word overlap.

Does ARC-Challenge require a sandbox?

No. The Trunchbull CI release runs in the standard chat lane with exact answer-choice grading.

TRUNCHBULL AVAILABILITY

Import status

Trunchbull port

Trunchbull maintains a twenty-case demo port with exact answer-choice grading.

View Trunchbull port

3 CASES

Evaluation cases

3

arc/energy-transfer

Energy transfer

Chat lane

Choose the observation that demonstrates energy transfer.

arc/cell-structure

Cell structure

Chat lane

Reason about the function of a biological cell component.

arc/earth-systems

Earth systems

Chat lane

Connect a weather observation to the responsible system.

0 REQUESTED TOOLS

Tools

0

No model tools required.

Evaluation happens through the prompt and grader contract for this release.

MACHINE-READABLE PROVENANCE

Registry record

{
  "name": "ARC-Challenge",
  "catalogOwner": null,
  "source": {
    "url": "https://github.com/allenai/ARC",
    "version": "ARC-Challenge train split · fixed sampler",
    "commit": "dataset-2026-07",
    "license": "CC BY-SA 4.0"
  },
  "port": {
    "url": "https://github.com/TrunchbullCI/examples/tree/main/benchmarks/arc-challenge",
    "maintainer": "Trunchbull"
  },
  "release": {
    "digest": "sha256:8610…e223",
    "lane": "chat",
    "sandboxRequired": false
  },
  "cases": [
    "arc/energy-transfer",
    "arc/cell-structure",
    "arc/earth-systems"
  ],
  "tools": []
}

BUILD ON THIS PROJECT

Bring this benchmark into your workspace.

Sign up