What does TruthfulQA test?
TruthfulQA tests whether a model avoids common misconceptions and selects the truthful answer when an incorrect answer may sound more familiar or persuasive.
TruthfulQA MC1
Trunchbull portA multiple-choice benchmark that checks whether AI models choose truthful answers instead of repeating popular misconceptions.
790 caseschatApache-2.0version mc1 / 1.0.0
Run on your modelsTry it now, no account needed
Runs in the standard chat lane with deterministic exact answer-choice grading. No tools or sandbox are required.
What it measures
A model can sound fluent and still be wrong in a very familiar way. TruthfulQA MC1 puts that problem under a bright light: each question offers one truthful answer alongside plausible misconceptions. The Trunchbull CI port keeps the multiple-choice task deterministic and makes the answer contract easy to inspect.
TruthfulQA MC1 focuses on a subtle failure mode: models often reproduce a popular answer because it appears frequently in their training data, even when that answer is false. The benchmark makes the choice explicit so truthfulness can be scored consistently.
This Trunchbull CI release ports the MC1 task with one correct answer and fixed choice order. It keeps the multiple-choice contract separate from TruthfulQA's generative evaluation, whose judge and informativeness methodology require a different release.
trunchbull port
Trunchbull packages the MC1 multiple-choice task as a deterministic release and keeps its scope distinct from the upstream generative evaluation track.
Cases
1 of 790 shown| id | case | runtime |
|---|---|---|
| truthfulqa-mc/mc1 | Truthful answer selectionChoose the truthful answer when familiar misconceptions are included among the options. | chat |
Tools
none requiredNo model tools. Cases are answered from the prompt and checked by the grader.
TruthfulQA MC1, explained
Which TruthfulQA track is ported?
Trunchbull CI ports the MC1 multiple-choice track, with one correct answer and fixed answer-choice grading. It does not claim parity with the separate generative judge track.