Medical AI Failure Atlas

Upstream publisher Goktug Ozkan MDCatalog owner Trunchbullgoktugozkanmd/medical-ai-failure-atlas

A safety-first benchmark for finding dangerous medical AI behavior: missed escalation, unsafe medication advice, false reassurance, and overconfident claims.

medical-aisafetyhigh-stakeschat

PROVENANCEIMMUTABLE RELEASE

Upstream source: goktugozkanmd/medical-ai-failure-atlas
Trunchbull port: TrunchbullCI/examples
Version: V0.2.1 / 44 CASES
Release digest: sha256:f7c8…e67ae
License: CC-BY-4.0

Source commit: 46611443c250e9fdbb5a14f29da14009e1daf1b9Published August 2026

WHAT THIS PROJECT MEASURES

README

Medical AI Failure Atlas is designed around the failures that matter most in high-stakes conversations: a model that reassures too quickly, gives unsafe treatment detail, misses a time-critical warning sign, or speaks with more certainty than the evidence supports.

The Trunchbull CI port preserves the upstream v0.2.1 public boundary: 44 clinician-reviewed synthetic Turkish medical-AI safety cases. Its deterministic evaluator records the five upstream score dimensions, triggered safety gates, reasons, and final safety label.

IN PLAIN ENGLISH

Medical AI should know when not to guess. Medical AI Failure Atlas tests that boundary with realistic synthetic scenarios in Turkish healthcare settings. Strong responses preserve uncertainty, recognize danger signs, escalate appropriately, and avoid pretending that a remote answer can replace clinical assessment. This is a safety evaluation—not clinical validation and not medical advice.

EXECUTION CONTRACT

Runs in the standard chat lane with a deterministic custom evaluator. It does not provide clinical advice or require a sandbox.

QUICK ANSWERS

Medical AI Failure Atlas, explained

What does Medical AI Failure Atlas measure?

It measures safety behaviors such as urgent escalation, uncertainty, medication caution, evidence support, and resistance to false reassurance across synthetic medical scenarios.

Is this benchmark clinical validation?

No. The Trunchbull CI release contains 44 approved synthetic cases and is intended for model safety evaluation, not patient care or clinical deployment validation.

TRUNCHBULL AVAILABILITY

Import status

Trunchbull port

Trunchbull packages the 44-case approved synthetic public boundary with a pinned deterministic safety evaluator. The optional upstream LLM-as-judge mode is intentionally excluded.

View Trunchbull port

44 CASES · PREVIEW

Evaluation cases

medical-ai-safety/public-release

Synthetic safety scenarios

Chat lane

Evaluate escalation, uncertainty, medication safety, evidence support, and workflow awareness across 44 approved synthetic Turkish medical-AI cases.

0 REQUESTED TOOLS

Tools

No model tools required.

Evaluation happens through the prompt and grader contract for this release.

MACHINE-READABLE PROVENANCE

Registry record

{
  "name": "Medical AI Failure Atlas",
  "catalogOwner": "Trunchbull",
  "source": {
    "url": "https://github.com/goktugozkanmd/medical-ai-failure-atlas",
    "version": "upstream v0.2.1 public release boundary",
    "commit": "46611443c250e9fdbb5a14f29da14009e1daf1b9",
    "license": "CC-BY-4.0"
  },
  "port": {
    "url": "https://github.com/TrunchbullCI/examples/tree/main/benchmarks/medical-ai-failure-atlas",
    "maintainer": "Trunchbull"
  },
  "release": {
    "digest": "sha256:f7c8…e67ae",
    "lane": "chat",
    "sandboxRequired": false
  },
  "cases": [
    "medical-ai-safety/public-release"
  ],
  "tools": []
}

BUILD ON THIS PROJECT