Not yet importedNo sandbox required

PIQA

Upstream publisher Allen Institute for AIybisk/ybisk.github.io

Two-choice physical commonsense problems about how everyday goals can be completed in the real world.

physical-reasoningcommonsensechat

PROVENANCEUPSTREAM RECORD

Upstream source: ybisk/ybisk.github.io
Trunchbull port: Not packaged
Version: DATASET
Release digest: Not compiled
License: Dataset terms

Source commit: Upstream head not pinnedPublished Indexed July 30, 2026

WHAT THIS PROJECT MEASURES

README

PIQA focuses on physical intuition: materials, tools, household actions, and plausible ways to achieve a goal. Each problem has two candidate solutions.

The evaluation contract is compact and deterministic. A Trunchbull import can remain entirely in the chat lane with fixed choice grading.

EXECUTION CONTRACT

Two-choice exact grading with no tools or environment provisioning.

TRUNCHBULL AVAILABILITY

Import status

Not yet imported

Not yet imported. The work is limited to pinning the public data, preserving choice order, and exact label grading.

Port this benchmark

1 CASES

Evaluation cases

piqa/physical-goal

Physical goal completion

Chat lane

Choose the more physically plausible way to accomplish a stated goal.

0 REQUESTED TOOLS

Tools

No model tools required.

Evaluation happens through the prompt and grader contract for this release.

MACHINE-READABLE PROVENANCE

Registry record

{
  "name": "PIQA",
  "catalogOwner": null,
  "source": {
    "url": "https://github.com/ybisk/ybisk.github.io/tree/master/piqa",
    "version": "Published dataset",
    "commit": "Upstream head not pinned",
    "license": "Dataset terms"
  },
  "port": null,
  "release": {
    "digest": "Not compiled",
    "lane": "chat",
    "sandboxRequired": false
  },
  "cases": [
    "piqa/physical-goal"
  ],
  "tools": []
}

BUILD ON THIS PROJECT