What happens when the models everyone keeps calling "doctor-level" have to actually take the test? We put that claim under pressure with a study that lands uncomfortably close to real life: six popular large language models were handed 137 multiple-choice questions on colorectal cancer and forced into a strict zero-shot format — no examples, no reasoning shown, just the letter. Performance came in around chance, or below. In other words, the models often sound certain while effectively guessing.

We break down why colorectal cancer is such a revealing stress test for medical AI. Guidelines evolve, screening recommendations shift, and a single case can move across primary care, GI, surgery, pathology, oncology, and radiation — so "knowing" this disease means tracking a moving, country-specific target, not reciting one fixed fact.

Then we turn to what these systems are really doing under the hood — next-token prediction, instruction tuning, reasoning-style layers — and why none of that guarantees a reliable guideline lookup. We walk through the failure modes that matter for patient safety: basic fact-retrieval errors, hierarchical logic breaking down in cancer staging, and hallucinations that could spawn unnecessary tests or wrong recommendations.

For clinicians, trainees, and curious patients leaning on chatbots for health questions, the takeaway is blunt: don't trust an answer that can't show its work and cite the guideline.

References:

Performance of next-generation AI chatbots in colorectal cancer knowledge assessment: a comparative pilot study of ChatGPT-5.1, Gemini-3Pro Preview, DeepSeek-V3.2, Kimi K2 Thinking, Qwen3-Max and Claude Opus 4.5
Chen et al.
Updates in Surgery (2026)

Credits:

Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)
 Licensed under Creative Commons: By Attribution 4.0
 https://creativecommons.org/licenses/by/4.0/