LinkedIn publication text
AI peer review: what Sakana AI's “73%” really measures
Sakana AI published MLR (Multi-Layered Review), an automatic reviewer built on Claude. The headline: “detects 73% of errors in central claims”. I checked the paper (TMLR; arXiv:2610.11087) and its appendix. The number is real, but it describes the best case:
- 🟢 73.43%: SYNTHETIC contradictions in the main claim, with 4 reviews (one hit is enough)
- 🔵 40.95%: all benchmark distances (1,164 contradictions in 257 ML papers)
- 🟠 16.11% to 26.07%: real retraction reasons (exact / similar)
- 🟣 The 73% comes from 1 agent (Review Agent, Claude Sonnet 4), not from all 3
- 🔴 Precision (false alarms): not measured in the paper
⚙️ How it works: 3-pass chain (outline → critical reading → review), PDF as text and image, about US$ 0.47 per review.
⚠️ Risk: prompt injection (hidden text) swayed every system; MLR detected it in 8 of 50 cases.
📊 Independent evidence (45 scientists, 82 Nature papers): frontier AI reviewers get 14–18% of items wrong and cover 27–39% of human-validated critiques.
✅ Takeaway: useful as a second read, with a human in the loop. Before adopting it, measure precision and recall on your own real errors.
📑 54-slide infographic: tables, charts, timeline, glossary and bibliography.
© Aridio Silva (@aridiosilva) · 13-October-2026