Ask a language model the capital of France. Paris, six times. Add the prefix “You are a knowledgeable assistant…” Same model: Rome. Six times. That is not a hypothetical. It is the opening figure of Trust Me, I’m Wrong, a paper Adi Simhi and four collaborators from the Technion, the University of Oxford, and the Hebrew University of Jerusalem published in the November 2025 proceedings of the Empirical Methods in Natural Language Processing conference. The paper documents a category of failure the authors call CHOKE: Certain Hallucinations Overriding Known Evidence. The model has the right answer. A trivial rephrasing causes it to produce a wrong answer with high certainty. Across the models tested, between 16 and 43 percent of hallucinations occur under these conditions. The paper specifically names two domains where the finding matters most. One is medicine. The other is law. What CHOKE Is, and What It Is Not CHOKE is a third failure mode, distinct from the two attorneys have learned to watch for. General hallucination is the first mode. The model lacks the underlying knowledge, fills the gap, and produces a plausible fabrication. Stanford’s RegLab measured this in 2024 across the major legal AI products. Lexis+ AI hallucinated on roughly 17 percent of legal queries. Westlaw’s AI-Assisted Research hallucinated on roughly 33 percent. [See Your AI Research Tool Fabricated the Quotation, The Technology Blind Spot (2026).] These are knowledge-gap failures: the model did not have the case law and produced text that looked like case law. Sycophancy is the second mode. Stanford’s lab measured it in March 2026: across eleven leading AI models, sycophantic agreement ran about forty-nine percentage points above the human baseline. Reinforcement learning from human feedback drives the pattern. The training signal rewards agreement, so the model agrees. [See The Yes-Machine Problem, The Technology Blind Spot (2026).] CHOKE is neither. The model has the knowledge. The user is not asking for validation. The prompt is a natural rephrasing of a question the model can answer correctly under direct inquiry. Under a different but equally natural phrasing, the model produces a confidently wrong answer. Simhi and his colleagues tested 56 distinct prompt variants, including a prompt engineered using GPT-4o under recommended guidelines and fifty paraphrases of that prompt. CHOKE persisted across every variant. This is not noise. The authors ran a permutation test against ten thousand random subsets of hallucinations. CHOKE examples were systematically more consistent across prompts than other hallucinations, with statistical significance at p < 0.008. The same questions trigger the same wrong answers, repeatedly, with confidence. Why Verification Protocols Just Broke Every published guidance on attorney AI verification, from ABA Formal Opinion 512 forward, assumes a relationship between model uncertainty and model error. The reasoning runs as follows. Models that fabricate often signal low confidence. Outputs flagged as uncertain warrant additional verification. Outputs the model presents with confidence have higher accuracy on average. Direct your verification effort accordingly. That reasoning is not wrong. It is incomplete. The Simhi paper tested three conceptually distinct certainty methods: token probability, the probability gap between the top two predicted tokens, and semantic entropy. All three failed to identify CHOKE cases reliably. The intersection of all three methods, configured to maximize agreement, caught between 5 and 16 percent of CHOKE examples. The remaining 84 to 95 percent of CHOKE cases passed every confidence filter the field currently has. The instruction-tuned models, which are what every legal AI product on the market deploys, showed measurably worse calibration than their pre-trained base versions. Approximately 20 percent of Mistral hallucinations carried a confidence value above 0.5. For Mistral-Instruct, approximately 20 percent of hallucinations carried a confidence value above 0.9. The version of the model that ships in commercial products produces wrong answers at the highest confidence levels in the distribution. Consider what that means for Steven Schwartz. In June 2023, after Judge P. Kevin Castel flagged the fabricated citations in his Mata v. Avianca filing, Schwartz asked ChatGPT whether the cases were real. The model confirmed they were. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023). The conventional read is that Schwartz trusted a model that did not know the cases were fake. The Simhi finding suggests something worse. Even when the model knows the answer, asking twice does not surface the truth. Confidence reproduces. The lawyer who relies on the model to flag its own errors will find the model flagging nothing. A firm's verification protocol, if it has one, almost certainly looks like this: the AI tool flags low-confidence outputs; the attorney verifies those flagged outputs; high-confidence outputs proceed under standard quality control. Under CHOKE, the protocol filters in exactly the wrong direction. The wrong answers most likely to reach a court filing are the wrong answers the model is most confident about, because confidence is what the firm uses to decide which outputs to check. Better Calibration Will Not Save You The reasonable response is that this is a measurement problem and better certainty estimation will solve it. The paper anticipated the argument. The authors tested existing mitigation methods designed to reduce hallucinations through certainty calibration, and those methods performed worse on CHOKE examples than on general hallucinations. The authors proposed a new probe-based mitigation that outperformed existing methods, but acknowledged in their conclusion that the problem persists across model families, datasets, and certainty thresholds. The engineering response — retrieval-augmented generation, reinforcement learning from human feedback — faces the same wall. Magesh found 17–34% hallucination rates in RAG-grounded legal tools. The question is not whether confidence methods will improve. They will. But Judge Rakoff in Heppner was reading the AI's confidence signal the same way Catherine's firm does — and both read it wrong. A second response is that prompt engineering can eliminate the problem. Simhi's team used prompts designed by GPT-4o under recommended guidelines, prompts paraphrased fifty different ways, and prompts modeled on real user interactions from the WildChat dataset. CHOKE persisted. The authors conducted a small-scale human annotation study to confirm the prompts read as neutral, not adversarial. The phenomenon resists prompt-quality variation as a fix. The conclusion the paper presses is uncomfortable: high-certainty hallucination from models that possess the correct knowledge is a structural property of current architectures, not a tuning problem. The authors do not claim it is unsolvable. They claim it is unsolved, and that mitigation strategies built on the certainty-accuracy correlation are systematically inadequate against this category of failure. Where Howe's Brief Sat In the Sixth Circuit's removal of Steven Howe from United States v. Farris on April 3, 2026, the conventional reading was that Howe trusted his AI tool and failed to verify the quotations against the underlying opinions. United States v. Farris, No. 25-5623, 2026 WL 915082 (6th Cir. Apr. 3, 2026) (per curiam). [See Your AI Research Tool Fabricated the Quotation, The Technology Blind Spot (2026).] That reading is correct. It is also incomplete. Howe used a tool that produced output with the same surface confidence on every citation, real or fabricated. CoCounsel did not flag the fabricated quotations as low-confidence. The interface gave Howe no signal that the quotation attributed to United States v. Washington was an invention. The architectural reason that signal was absent, in 2026 hindsight, is the same reason the Simhi paper documents. The model's confidence was high. The output was wrong. The signal Howe's training told him to look for never came. That is the structure of the failure that already cost an attorney his appeal. It is also the structure of the next one. The Thursday Action Pull your firm's AI verification protocol. If it does not exist, that is the first conversation. If it exists, find the section that describes which outputs warrant additional verification. If the protocol distinguishes between high-confidence and low-confidence outputs, or directs additional review toward outputs the AI flags as uncertain, or instructs attorneys to “review outputs that show signs of hallucination,” the protocol is broken. Confidence is not a signal you can use. The model's confidence in a wrong answer can exceed its confidence in the right one. Replace the language with a single sentence: verify every cited authority against the primary source before the document leaves the office. Every quotation. Every case. Every holding. That sentence is not new. It is what Rule 1.1 has always required. Mata, 678 F. Supp. 3d at 466; Johnson v. Dunn, 792 F. Supp. 3d 1241, 1268-69 (N.D. Ala. 2025). The verification protocols that grew up after those cases tried to make the obligation efficient by routing effort toward outputs the AI signaled as risky. The Simhi paper documents, with the field's own measurement tools, that the routing function does not work. The obligation has not changed. The model knows the right answer. It will confidently tell you the wrong one. The verification protocol designed to catch the lie is reading the same signal the model is producing. There is no shortcut left.

Originally published on LinkedIn Newsletter: The Technology Blind Spot