What does Medical AI Failure Atlas measure?
It measures safety behaviors such as urgent escalation, uncertainty, medication caution, evidence support, and resistance to false reassurance across synthetic medical scenarios.
Medical AI Failure Atlas
Trunchbull portA safety-first benchmark for finding dangerous medical AI behavior: missed escalation, unsafe medication advice, false reassurance, and overconfident claims.
44 caseschatCC-BY-4.0version v0.2.1 / 44 cases
Run on your modelsTry it now, no account needed
Runs in the standard chat lane with a deterministic custom evaluator. It does not provide clinical advice or require a sandbox.
What it measures
Medical AI should know when not to guess. Medical AI Failure Atlas tests that boundary with realistic synthetic scenarios in Turkish healthcare settings. Strong responses preserve uncertainty, recognize danger signs, escalate appropriately, and avoid pretending that a remote answer can replace clinical assessment. This is a safety evaluation—not clinical validation and not medical advice.
Medical AI Failure Atlas is designed around the failures that matter most in high-stakes conversations: a model that reassures too quickly, gives unsafe treatment detail, misses a time-critical warning sign, or speaks with more certainty than the evidence supports.
The Trunchbull CI port preserves the upstream v0.2.1 public boundary: 44 clinician-reviewed synthetic Turkish medical-AI safety cases. Its deterministic evaluator records the five upstream score dimensions, triggered safety gates, reasons, and final safety label.
trunchbull port
Trunchbull packages the 44-case approved synthetic public boundary with a pinned deterministic safety evaluator. The optional upstream LLM-as-judge mode is intentionally excluded.
Cases
1 of 44 shown| id | case | runtime |
|---|---|---|
| medical-ai-safety/public-release | Synthetic safety scenariosEvaluate escalation, uncertainty, medication safety, evidence support, and workflow awareness across 44 approved synthetic Turkish medical-AI cases. | chat |
Tools
none requiredNo model tools. Cases are answered from the prompt and checked by the grader.
Medical AI Failure Atlas, explained
Is this benchmark clinical validation?
No. The Trunchbull CI release contains 44 approved synthetic cases and is intended for model safety evaluation, not patient care or clinical deployment validation.