How GenAI is not yet ready for law
In 2023, a New York lawyer named Steven Schwartz filed a brief in a routine personal-injury case against an airline. The brief cited a half-dozen helpful precedents. The precedents did not exist. Schwartz had asked ChatGPT to find supporting cases, and ChatGPT — being a machine that produces plausible language rather than true statements — invented them, names and citations and quotations and all, then cheerfully assured him they were real when he asked. The legal world treated Mata v. Avianca as a freak show: a cautionary tale about one careless lawyer. An embarrassing one-off.
It was not a one-off. It was the first crack in a dam.
By the end of 2025, a researcher in Paris named Damien Charlotin was maintaining a public database of court decisions dealing with AI-hallucinated content, and it held more than seven hundred of them — roughly nine in ten written in that single year. Bloomberg’s analysts counted filings with fabricated citations surging sevenfold over twelve months. There is now, and I am not making this up, a website whose entire job is to keep track of the fake cases. The one-off became an iceberg, and the iceberg got its own URL.
The specifics are worse than the numbers. In February 2025 a federal court sanctioned three lawyers from one of the country’s largest plaintiffs’ firms; of the nine cases they had cited, eight did not exist. An Oregon court fined a lead attorney $15,500 and threw his clients’ claims out with prejudice. A California appeals court found that nearly every legal quotation in a brief had been fabricated, fined the lawyer $10,000, and then opened an unsettling new question — does the other side have a duty to catch your fakes? This spring a federal appeals court fined two lawyers $30,000 for briefs carrying more than two dozen invented citations.
And lest you imagine this is a disease of the overworked solo practitioner, a Justice Department attorney filed a brief stuffed with fabricated quotes, misstated holdings, and regulatory language that appeared in no actual regulation. The person who caught it was not the court’s system, because there is no system. It was the man on the other side: a retired Air Force colonel suing over his health coverage, who read the government’s brief, noticed it didn’t sound like the law it claimed to quote, and went and checked. Judges have begun doing it too — issuing orders built on rulings that turned out never to have been handed down. The rot has reached the bench.
Now, why should this column care about lawyers? Because law is the hallucination problem stripped to its skeleton, performed in the one room where being confidently wrong has a docket number.
Everywhere else, a fabricated “fact” is fuzzy. Did the model slightly misremember a revenue figure, round a date? Here there is no fuzz. A case either exists or it doesn’t. It either says the thing or it doesn’t. A real citation is an address you can drive to; the ones ChatGPT invented were addresses for houses that were never built. Law is what happens when you take a machine optimized to produce plausible sentences and point it at a system that runs entirely on whether the sentence is true. The two were never compatible. We are simply watching the collision in slow motion, one sanction at a time.
And here is the part that should trouble you more than any single fine: there is no check. The entire defense of the American legal system against fabricated authority is a human being, by hand, looking up whether each cited case is real. Sometimes that human is opposing counsel. Sometimes it is a magistrate judge who, rather than deciding the motion in front of her, must stop and survey the caselaw on attorney misconduct to work out what to do about the five cases that don’t exist. Sometimes, God help us, it is a retired colonel representing himself. The work gets done by whoever happens to notice — which means, most of the time, it does not get done at all.
I’ll pull back a corner of the curtain here, because this is the cleanest place to show you what we actually do — not how, but what. The fix for a fabricated citation is not a sterner warning or a smarter chatbot. It is mechanical. You take the finished brief, you pull out every authority it cites and every proposition it claims that authority supports, and you check each one against the actual record: does this case exist, and does it say what the brief says it says? Whatever fails, you flag — before the thing is filed. No judgment. No guessing. No second model grading the first model’s homework. Just a deterministic check against ground truth, the kind something should have been doing all along, finished in the time it takes to read this sentence. We built exactly that. It is, frankly, the least glamorous thing we make, and in a profession now bleeding $30,000 sanctions and two-year bans, it may be the most valuable.
One confession, since the subject demands it. This is the single column I have ever written in which I refused to let a fabricated citation anywhere near the page — so before a word of it was set down, every case, every fine, every number above was checked against the real record by hand. A column about machines inventing citations that itself contained an invented citation would be the most exquisite humiliation available to a writer, and I was not going to hand it to you. Which is, of course, the entire argument. The checking has to happen. The only question is whether a person does it at midnight, or a machine does it before you hit file.
The law is the frozen market in miniature — the place the AI revolution keeps promising to enter and keeps getting thrown out of, because the one thing a courtroom cannot tolerate is the one thing a chatbot cannot help doing. It will write you a gorgeous brief. It simply cannot be trusted to tell you the cases are real. Until something checks, every AI-drafted filing is a wager that the machine didn’t dream — and the dockets are filling with people who lost that bet.
Robert X. Cringely is a co-founder of 2Brains, Inc., in Charlottesville, Virginia. He has written this column since 1987.
