What does ARC-Challenge measure?
ARC-Challenge measures science knowledge and reasoning on difficult grade-school questions designed to resist shallow retrieval and word overlap.
ARC-Challenge
Trunchbull portA clear, no-frills test of science reasoning on challenging grade-school questions with multiple-choice answers.
3 case previewschatCC BY-SA 4.0version arc / train
Run on your modelsTry it now, no account needed
Runs in the standard chat lane with exact answer-choice grading.
What it measures
ARC-Challenge is built for questions that are harder than simple recall. Each case gives the model a science question and a fixed set of choices; the challenge is choosing the answer that follows from the underlying concept. Trunchbull CI packages that contract with pinned provenance and transparent grading.
ARC-Challenge is the difficult partition of the AI2 Reasoning Challenge. Each case presents a science question and a fixed set of answer choices.
Trunchbull exposes a small deterministic demo port. The grader extracts a final answer choice and compares it with the published answer key.
trunchbull port
Trunchbull maintains a twenty-case demo port with exact answer-choice grading.
Cases
3 in this release| id | case | runtime |
|---|---|---|
| arc/energy-transfer | Energy transferChoose the observation that demonstrates energy transfer. | chat |
| arc/cell-structure | Cell structureReason about the function of a biological cell component. | chat |
| arc/earth-systems | Earth systemsConnect a weather observation to the responsible system. | chat |
Tools
none requiredNo model tools. Cases are answered from the prompt and checked by the grader.
ARC-Challenge, explained
Does ARC-Challenge require a sandbox?
No. The Trunchbull CI release runs in the standard chat lane with exact answer-choice grading.