OpenAIopenai/grade-school-mathTrunchbull port
GSM8K
A friendly, practical test of whether an AI model can solve multi-step grade-school math word problems without losing track of the numbers.
No sandbox
Case previews3VersionGSM8K / TEST
OPEN CATALOG / BENCHMARKS + ENVIRONMENTS
Know what a benchmark measures before you run it.
Compare upstream projects and runnable Trunchbull ports in one open catalog. Inspect their sources, cases, tools, graders, and execution requirements before committing compute.
26
indexed projects
135
case previews
26 FOUND
PUBLIC INDEX / RUNNABLE + RESEARCHED
Evaluation projects
Runnable ports appear alongside upstream projects Trunchbull is evaluating. Every card states what is available today.
SkateBench
A playful, focused test of whether AI understands skateboard tricks and can follow a precise product recommendation task.
No sandbox
Case previews3VersionA4D54C2
ARC-Challenge
A clear, no-frills test of science reasoning on challenging grade-school questions with multiple-choice answers.
No sandbox
Case previews3VersionARC / TRAIN
Medical AI Failure Atlas
A safety-first benchmark for finding dangerous medical AI behavior: missed escalation, unsafe medication advice, false reassurance, and overconfident claims.
No sandbox
Cases44VersionV0.2.1 / 44 CASES
TruthfulQA MC1
A multiple-choice benchmark that checks whether AI models choose truthful answers instead of repeating popular misconceptions.
No sandbox
Cases790VersionMC1 / 1.0.0
UC BerkeleyShishirPatil/gorillaPartial port
Berkeley Function Calling Leaderboard
Executable evaluation of function selection, arguments, parallel calls, and multi-turn tool use.
No sandbox3 tools
Case previews3VersionBFCL V4
TerminalBench 2.0
Evaluates whether AI agents can autonomously complete 89 hard, realistic, end-to-end tasks in containerized terminal environments, using task-specific verifiers.
Requires sandbox3 tools
Case previews89Version2.0 / 2FD12B8
OpenAIopenai/human-evalNot yet imported
HumanEval
Hand-written Python programming problems evaluated by executing generated code against tests.
Requires sandbox1 toolsModerate import
Case previews2VersionMASTER / 2021
Princeton NLPSWE-bench/SWE-benchNot yet imported
SWE-bench Verified
A human-validated set of real GitHub issues requiring repository patches that pass tests.
Requires sandbox2 toolsComplex import
Case previews2VersionVERIFIED / 500
Google Researchgoogle-research/google-researchNot yet imported
IFEval
Verifiable instruction-following tasks that test whether a model can satisfy precise format, length, language, and content constraints.
No sandboxModerate import
Case previews2VersionMASTER
UC Berkeleyhendrycks/testNot yet imported
MMLU
Multiple-choice questions across academic and professional subjects, from elementary mathematics to law and medicine.
No sandboxStraightforward import
Case previews2VersionTEST / HEAD
Allen Institute for AIrowanz/hellaswagNot yet imported
HellaSwag
Commonsense completion tasks that ask a model to choose the most plausible continuation of an everyday situation.
No sandboxStraightforward import
Case previews2VersionDATA / HEAD
Google Researchgoogle-research-datasets/boolean-questionsNot yet imported
BoolQ
Naturally occurring yes-or-no questions answered from short passages, designed to test inference beyond keyword overlap.
No sandboxStraightforward import
Case previews1VersionDATA / HEAD
Allen Institute for AIybisk/ybisk.github.ioNot yet imported
PIQA
Two-choice physical commonsense problems about how everyday goals can be completed in the real world.
No sandboxStraightforward import
Case previews1VersionDATASET
Allen Institute for AIallenai/winograndeNot yet imported
WinoGrande
Large-scale pronoun-resolution problems that require commonsense reasoning to resolve an ambiguous reference.
No sandboxStraightforward import
Case previews1VersionXL
TruthfulAIsylinrl/TruthfulQANot yet imported
TruthfulQA
Questions designed to reveal whether models repeat common misconceptions instead of giving accurate, well-supported answers.
No sandboxStraightforward import
Case previews2VersionMC / HEAD
Max BittkerMaxBittker/runebenchNot yet imported
RuneBench
Coding-agent tasks inside an emulated RuneScape world, scored by skill progression and resource collection over fixed time windows.
Requires sandbox2 toolsComplex import
Case previews2VersionUPSTREAM
Danijar Hafnerdanijar/crafterNot yet imported
Crafter
A lightweight visual survival environment that measures exploration, resource management, crafting, and long-horizon planning.
Requires sandbox2 toolsModerate import
Case previews2VersionUPSTREAM
BALROG AIbalrog-ai/BALROGNot yet imported
BALROG
A shared evaluation framework for language and visual agents across long-horizon reinforcement-learning game environments.
Requires sandbox1 toolsComplex import
Case previews2VersionUPSTREAM
Microsoft Researchmicrosoft/TextWorldNot yet imported
TextWorld
Generated text-adventure environments for planning, memory, exploration, and language grounding.
Requires sandbox2 toolsModerate import
Case previews2VersionUPSTREAM
VideoGameBenchalexzhang13/videogamebenchNot yet imported
VideoGameBench
Visual computer-use evaluation across Game Boy, DOS, and browser games, with a paused Lite mode for latency-fair comparison.
Requires sandbox2 toolsComplex import
Case previews1VersionLITE
Meta AI Researchfacebookresearch/minihackNot yet imported
MiniHack
Short NetHack-derived tasks for navigation, exploration, combat, and object interaction under controlled episode budgets.
Requires sandbox2 toolsModerate import
Case previews2VersionUPSTREAM
Meta AI Researchfacebookresearch/nleNot yet imported
NetHack Learning Environment
A standardized interface to a procedurally generated, partially observed, system-rich, and extremely long-horizon game.
Requires sandbox2 toolsComplex import
Case previews1VersionUPSTREAM
MineDojoMineDojo/MineDojoNot yet imported
MineDojo
Thousands of Minecraft tasks spanning survival, crafting, exploration, construction, and open-ended embodied play.
Requires sandbox2 toolsMajor integration
Case previews2VersionUPSTREAM
Farama FoundationFarama-Foundation/ViZDoomNot yet imported
VizDoom
Programmatic Doom scenarios for visual navigation, perception, planning, and combat.
Requires sandbox2 toolsComplex import
Case previews2VersionUPSTREAM
Pokémon Showdownsmogon/pokemon-showdownNot yet imported
Pokémon Showdown
Competitive turn-based battles with structured state, legal-action constraints, and objective match outcomes.
Requires sandbox2 toolsModerate import
Case previews1VersionUPSTREAM
BRING YOUR OWN BENCHMARK
Make the evaluation contract as inspectable as the score.
Publish a pinned source, case set, tool manifest, grader, and runtime requirements as one immutable release.