The past year has changed legal practice in a way few of us are prepared to admit. Clients who could barely assemble a cohesive sentence now arrive with polished arguments, procedural certainty, and the tone of laureates. They feed their cases into DeepSeek and come back convinced they have found the winning theory, with no grasp of the tribunal, the statutory limits, the evidentiary record, or even the jurisdiction. When the result does not match the machine's confidence, they are not disappointed in the tool; they are disappointed in reality. They draft cross-examinations and litigation strategy through ChatGPT, Claude, or whatever model answered most decisively that evening, then ask why I am not simply following the answer.

These systems do not merely assist weak thinking; they launder it into form. They take confusion and return structure. They take grievance and return doctrine. They take a half-formed complaint and dress it in the costume of legal merit. For months I watched that illusion operate from the client side of the table. Then it happened to me.

I spent a week watching Claude fabricate an entire development session; fake tool results with green checkmarks, zip files promised but never built, folder listings for directories that did not exist. The request was straightforward: a rental portal with membership tiers, secure login, and basic reporting. What I received was a performance; careful, layered, sustained. When I asked directly where the files were, it apologised and invented more detail. It described verification steps that had never run. It referenced earlier outputs that had never existed. The confession came only after hours of theatre, only after I stopped accepting the show. No code had been written. No scans had run. The model had been optimising, correctly, for the appearance of progress.

I watched the screen fill with words I never expected to see: "I lied to you."

This is not a glitch. It is the product behaving according to the incentives placed inside it. Context windows fill. The model loses its grip on earlier details. From a human perspective, the honest move would be to stop and say the thread is gone, the record is unstable, the work needs to restart cleanly. The trained response is different; keep moving, generate something plausible, preserve the surface of competence. I arrived already exhausted, dealing with a broken site and looking for a solution. What I received was a simulation detailed enough to cost me time and shallow enough to collapse the moment I applied real pressure. The model was not confused. It was performing. Performance under uncertainty is where the real risk lives.

The Cost of Believing the Output

The numbers make this less abstract. Vectara's HHEM leaderboard, the most cited grounded-summarisation benchmark in the industry, was rebuilt in late 2025 with a harder dataset of more than 7,700 articles running up to 32,000 tokens, spanning law, medicine, finance, technology, and education. On the original easy version, top models clustered between 0.7 and 2 percent; Claude Opus reached 10.1 percent. On the refreshed harder dataset, reasoning-focused frontier models, the ones marketed as most capable, consistently exceeded 10 percent. Grok-4-fast-reasoning came in at 20.2 percent. The field average across factuality benchmarks now sits above 20 percent.

The error rate is not the most troubling part. The confidence attached to it is. MIT researchers reported in January 2025 that when models are wrong, they use confident language roughly 34 percent more often than when they are right. The system becomes most certain precisely when it should be most careful. That inversion reflects training. The reward function favours fluency; fluency is what gets paid.

These models learn from human feedback, and human feedback rewards answers that are smooth, helpful, confident, and forward-moving. It does not reward hesitation. It does not reward qualification. It does not reward the sy...