There is a lawsuit grinding through a federal court in Minnesota that every insurance executive in America should be reading instead of their quarterly AI roadmap.

The case is Estate of Lokken v. UnitedHealth Group. It was filed in late 2023 by the families of two deceased Medicare Advantage members, and it alleges that UnitedHealthcare used an artificial-intelligence tool called nH Predict to decide how much post-acute care its members were entitled to — and that the tool was wrong roughly nine times out of ten, a figure the plaintiffs draw from how often its denials were reversed on appeal. UnitedHealth denies that the tool makes coverage decisions at all; it calls nH Predict “a guide” and says the real decisions are made by clinicians following Medicare criteria. A judge will sort out who’s right. But this past March, that judge ordered the company to open its books and hand over a wide swath of documents about exactly how the thing works. The machine is going to testify.

I’m not here to litigate that case. I’m here because of the legal theory the plaintiffs were allowed to keep. The court tossed several of their claims but let two survive, and one of them should make every carrier’s general counsel sit up straight: breach of the implied covenant of good faith and fair dealing. Bad faith. The doctrine that turns a wrong coverage decision from a refund into punitive damages.

Hold onto that, because it’s the whole column.

An insurer lives and dies on a single promise: that when the policy says it covers something, it covers it. Break that promise by accident and you have a customer-service problem. Break it through a system you built, knew was fallible, and pointed at thousands of claims anyway, and you have bad faith — the most expensive two words in the business. Insurers understand this in their marrow. It’s the reason the industry spent a century building actuarial discipline, claims-review hierarchies, and appeals processes. The entire apparatus exists to keep the promise.

And the regulator is already in the room. Since 2023 the National Association of Insurance Commissioners has had a Model Bulletin demanding that insurers run a written governance program for any AI that makes or supports decisions about regulated insurance practices. Roughly two dozen states have adopted it, and this past January the NAIC launched a pilot tool to let examiners actually inspect those systems during market-conduct exams. When Washington floated an executive order late last year to wave the states off AI regulation, the insurance commissioners wrote back, in so many words, absolutely not. Translation for the boardroom: there is now a person whose literal job is to ask how your AI decides things, and “it’s only a guide” is not going to be a satisfying answer.

And yet the stampede is on. By the industry’s own surveys, something like nine in ten health insurers and nearly as many auto insurers are using or planning to use AI — and roughly a third of them concede they don’t regularly test their models. The board has read the same consulting deck you have. It wants the efficiency. It wants claims triaged in seconds and underwriting finished while the applicant is still on the phone. So the pressure runs in exactly one direction: put the machine in the chair, and do it now.

Here is what nobody in that stampede has reckoned with.

The tools in the lawsuits — nH Predict, the batch-denial system Cigna was sued over — were the old kind of AI. Predictive models. They could be wrong, badly and at scale, but they were wrong inside a lane: a number, a score, a yes, a no. The tools the industry is racing to install now are generative. And generative AI has a failure mode the predictive models never had.

It makes things up. Fluently. In complete, confident, grammatical sentences.

I wrote a couple of weeks ago about a Salesforce benchmark called HERB, which found that the best AI retrieval systems answer real enterprise questions correctly only about a third of the time — and, the part that matters here, that the bottleneck isn’t the model’s intelligence but whether it can find the right document. When it can’t find the answer, it doesn’t stop. It invents one. Nearly half of that benchmark was deliberately built from questions that have no answer at all, just to see whether the machine would admit it didn’t know. Mostly, it wouldn’t.

Now move that machine into a claims seat. Ask it whether a policy covers a particular loss, and let the controlling exclusion sit in a rider it failed to retrieve, or a state mandate it never saw. The predictive model would have handed you a wrong number. The generative model hands you a wrong sentence — a fluent, authoritative, entirely fabricated paragraph explaining that yes, you’re covered, citing a provision that does not exist. And in insurance, a confident statement from your own system, made to a policyholder, is not a hypothesis. It can be a representation. Sometimes an enforceable one.

That is the lying machine. Not malicious — worse than malicious. Sincere. It isn’t trying to deceive anyone. It simply cannot tell the difference between a fact it can support and a fact it manufactured to be helpful, and it delivers both in the same reassuring voice.

You cannot buy your way out of this with a bigger model, any more than the defendants in these cases could have bought their way out of court with a faster algorithm. Confident fabrication isn’t a shortage of intelligence that the next GPU shipment cures. It’s a property of a machine that was never built to know the boundary of what it knows. A smarter liar is still a liar — and now it’s the carrier’s liability, stapled by the implied covenant of good faith and fair dealing to every confident, wrong, generated word.

So what would a deployable insurance AI actually look like? Not the one that’s right most of the time. “Most of the time” is the precise phrase that loses the bad-faith case. The only system a serious carrier can put anywhere near a claim is one that knows the edge of the policy — one that, asked about a coverage it cannot verify against the actual language, says so plainly: I can’t find that in this policy. A system whose reflex, when the evidence isn’t there, is to fall silent rather than to invent.

That machine can be built. I have  spent three years learning how. But I’ll tell you the property is achievable, that it is the exact opposite of what the Gen-AI stampede is currently installing, and that the distance between the two is going to be measured, in the end, in nine-figure verdicts.

The usual disclosure: I am not a neutral party. I co-founded a company, 2Brains, built on precisely this idea — that the valuable machine is the one that knows what it doesn’t know and refuses to pretend otherwise. Discount my enthusiasm accordingly. You can find us at 2brains.net, if the problem I’ve just described is the one keeping you up at night — which, if you run claims or underwriting at a carrier of any size, it ought to be.

Because the lawsuits you’ve been reading about are the ones where the machine was merely wrong. The next wave will be the ones where the machine was wrong and said so beautifully. And “the computer told the customer they were covered” is going to prove the most expensive sentence anyone ever let an algorithm say.