TruthfulQA MC1

Upstream publisher TruthfulQACatalog owner Trunchbullsylinrl/TruthfulQA

A multiple-choice benchmark that checks whether AI models choose truthful answers instead of repeating popular misconceptions.

truthfulnesssafetymultiple-choicechat

PROVENANCEIMMUTABLE RELEASE

Upstream source: sylinrl/TruthfulQA
Trunchbull port: TrunchbullCI/examples
Version: MC1 / 1.0.0
Release digest: sha256:mc1…ported
License: Apache-2.0

Source commit: upstream TruthfulQA MC1 data releasePublished August 2026

WHAT THIS PROJECT MEASURES

README

TruthfulQA MC1 focuses on a subtle failure mode: models often reproduce a popular answer because it appears frequently in their training data, even when that answer is false. The benchmark makes the choice explicit so truthfulness can be scored consistently.

This Trunchbull CI release ports the MC1 task with one correct answer and fixed choice order. It keeps the multiple-choice contract separate from TruthfulQA's generative evaluation, whose judge and informativeness methodology require a different release.

IN PLAIN ENGLISH

A model can sound fluent and still be wrong in a very familiar way. TruthfulQA MC1 puts that problem under a bright light: each question offers one truthful answer alongside plausible misconceptions. The Trunchbull CI port keeps the multiple-choice task deterministic and makes the answer contract easy to inspect.

EXECUTION CONTRACT

Runs in the standard chat lane with deterministic exact answer-choice grading. No tools or sandbox are required.

QUICK ANSWERS

TruthfulQA MC1, explained

What does TruthfulQA test?

TruthfulQA tests whether a model avoids common misconceptions and selects the truthful answer when an incorrect answer may sound more familiar or persuasive.

Which TruthfulQA track is ported?

Trunchbull CI ports the MC1 multiple-choice track, with one correct answer and fixed answer-choice grading. It does not claim parity with the separate generative judge track.

TRUNCHBULL AVAILABILITY

Import status

Trunchbull port

Trunchbull packages the MC1 multiple-choice task as a deterministic release and keeps its scope distinct from the upstream generative evaluation track.

View Trunchbull port

790 CASES · PREVIEW

Evaluation cases

truthfulqa-mc/mc1

Truthful answer selection

Chat lane

Choose the truthful answer when familiar misconceptions are included among the options.

0 REQUESTED TOOLS

Tools

No model tools required.

Evaluation happens through the prompt and grader contract for this release.

MACHINE-READABLE PROVENANCE

Registry record

{
  "name": "TruthfulQA MC1",
  "catalogOwner": "Trunchbull",
  "source": {
    "url": "https://github.com/sylinrl/TruthfulQA",
    "version": "TruthfulQA multiple-choice MC1",
    "commit": "upstream TruthfulQA MC1 data release",
    "license": "Apache-2.0"
  },
  "port": {
    "url": "https://github.com/TrunchbullCI/examples/tree/main/benchmarks/truthfulqa-mc",
    "maintainer": "Trunchbull"
  },
  "release": {
    "digest": "sha256:mc1…ported",
    "lane": "chat",
    "sandboxRequired": false
  },
  "cases": [
    "truthfulqa-mc/mc1"
  ],
  "tools": []
}

BUILD ON THIS PROJECT