What does GSM8K measure?
GSM8K measures multi-step mathematical reasoning on grade-school word problems. It is especially useful for spotting arithmetic mistakes, missed quantities, and broken chains of reasoning.
GSM8K
Trunchbull portA friendly, practical test of whether an AI model can solve multi-step grade-school math word problems without losing track of the numbers.
3 case previewschatMITversion gsm8k / test
Run on your modelsTry it now, no account needed
Runs in the standard chat lane with no tools or sandbox environment.
What it measures
GSM8K is a simple benchmark to understand and a useful one to run: give a model a word problem, let it work through the arithmetic, and check whether the final number is right. Trunchbull CI turns that familiar test into a reproducible release with visible grading.
GSM8K contains linguistically diverse grade-school math problems that usually require two to eight steps to solve. It is useful for measuring whether a model can preserve quantities and operations across a short chain of reasoning.
This Trunchbull release exposes a fixed public sampler with an exact numeric-answer grader. The upstream source remains authoritative; the release records the source version and evaluation contract used by the demo.
trunchbull port
Trunchbull maintains a runnable twenty-case demo port with a pinned numeric-answer grader.
Cases
3 in this release| id | case | runtime |
|---|---|---|
| gsm8k/natalia-clips | Natalia's clipsMulti-step addition after calculating a fractional value. | chat |
| gsm8k/robe-fabric | Fabric for robesUnit multiplication and subtraction across two quantities. | chat |
| gsm8k/duck-eggs | Duck eggsDaily production, household use, and revenue calculation. | chat |
Tools
none requiredNo model tools. Cases are answered from the prompt and checked by the grader.
GSM8K, explained
How is GSM8K graded on Trunchbull CI?
The Trunchbull release uses exact numeric-answer grading, so a result passes when its normalized final answer matches the published answer.