Automation Bias: A Thinking Model for AI Tools

In May 2023, a New York lawyer named Steven Schwartz filed a court brief in a case against Avianca airline. The brief cited six legal precedents — full case names, docket numbers, courts, years, and reasoning that tracked the argument cleanly.

None of the cases existed. ChatGPT had invented all six.

When the judge asked Schwartz to produce the full opinions, he couldn’t. The cases weren’t in any legal database because they’d never been decided. In his response to the court, Schwartz explained he “was unaware that ChatGPT could fabricate cases.” His filing partner had checked the citations by asking ChatGPT if they were real. ChatGPT confirmed they were.

This is not a story about incompetence. Schwartz had practiced law for decades. What happened to him has a name — one that predates generative AI by decades — and understanding it is the most useful calibration adjustment anyone using AI tools can make right now.

What automation bias actually is

The term was coined by researchers Kathleen Mosier and Linda Skitka in the 1990s,¹ studying pilots using automated cockpit systems. Their definition is precise: the tendency to use automation as a heuristic replacement for vigilant information seeking and processing. Their research also identified two distinct error types:

Errors of commission — you take an action you shouldn’t because the automated system recommended it, even when other indicators contradict it. Schwartz submitted citations he hadn’t verified because the AI produced them with the same formatting and confidence as real ones.

Errors of omission — you fail to act because the system didn’t alert you to a problem. The AI summary that leaves out a critical clause; you don’t catch what you were never shown.

Both errors share the same root: vigilance was outsourced to the system, and the system didn’t return it.

The groundwork was laid earlier by cognitive scientist Lisanne Bainbridge, who in 1983 described the “ironies of automation”²: the more reliable an automated system, the less likely a human operator is to notice when it fails — and the less capable they are of recovering when it does.

The canonical example is Air France Flight 447³, which crashed into the Atlantic in 2009 with 228 people aboard. The Airbus autopilot disengaged at 35,000 feet after ice blocked the speed sensors. The pilots, who had been monitoring the autopilot for hours, suddenly had to fly manually. They pulled up into a stall instead of leveling off. It took 54 seconds from the autopilot disengaging to the plane becoming unrecoverable. The investigation found that the pilots’ manual flying skills had atrophied from disuse. Their first instinct — under stress, with confusing instrument readings — was to trust what the system had been doing, not to think from scratch.

The flight had been fine, until it wasn’t. The fact that it had been fine was exactly why it failed.

Why AI tools change the equation

Autopilot gave pilots confidence because it had a strong track record. But autopilot’s failures were still detectable — instruments read wrong, alerts sounded, things clearly didn’t add up. The failure modes announced themselves.

AI tools are different because of fluency. The output doesn’t just perform a function — it performs expertise. It presents itself with the register and confidence of someone who knows what they’re talking about. A model asked to summarize a contract produces text that reads like a lawyer reviewed it. Asked to write code, it produces code that reads like an experienced engineer wrote it. Asked to find legal precedents, it produces case citations that look exactly like real case citations.

This breaks the heuristic most people use to calibrate trust. When a junior analyst gives a shaky response, the delivery usually signals the shaky reasoning. When an AI gives a shaky response, it reads with the same clarity as a correct one. The presentation doesn’t degrade with the quality of the reasoning underneath.

Schwartz didn’t fail to check because he was careless. He failed because the output gave him no signal that checking was necessary. That’s what fluency does — it decouples the appearance of reliability from actual reliability.

The domain problem

The other dimension is how well you know what you’re reviewing. And this is where automation bias plays out differently depending on your expertise.

In familiar territory, fluency is less dangerous. A backend engineer reviewing AI-generated API code can evaluate it on substance — the patterns are familiar, the wrong assumptions stand out, the tests he’d write cover the edge cases he knows to worry about. The presentation doesn’t fool him because he has a reference point for what right actually looks like.

In unfamiliar territory, fluency is almost all you have. A backend engineer reviewing AI-generated payment processing in a codebase she’s never touched. A product manager asking AI to summarize a technical spec she doesn’t fully understand. Someone reviewing an AI summary of a contract without legal training. In each case, the output looks right because they don’t have a strong prior for what right looks like. They’re evaluating presentation, not substance.

This is the trap: your ability to review output is domain-specific. The AI’s confident presentation is not. The same fluency appears whether the AI is on solid ground or in deep water. You don’t get a warning when you’ve crossed from one to the other.

There’s a version of this that catches developers specifically. The code compiles. Tests pass. The PR looks clean. What doesn’t surface in any of that is the subtle security vulnerability, the architectural choice that creates friction two features from now, the edge case in the error handling that production traffic will eventually find. Tests confirm behavior against a spec. They don’t confirm the spec was complete, or that the implementation has no implications beyond the spec.

“The tests pass” feels like enough. Often it is. The calibration question is knowing when it isn’t.

The calibration that actually helps

Automation bias is a calibration problem, not a trust problem. The issue isn’t that Schwartz trusted AI — it’s that he trusted it at the same level regardless of context. He trusted a ChatGPT citation the same way he’d trust a Westlaw result, without accounting for how differently each system works.

Three questions that change the default:

Can I verify this independently? Not “does it look right” — that’s the fluency check, which is exactly what automation bias exploits. The question is whether there’s an external reference. Westlaw for legal citations. A staging environment for code. The source document for an AI summary. If there’s no independent path, you’re extending trust on presentation alone.

What’s the cost of being wrong? A low-stakes internal note is different from a brief filed in federal court. A script against test data is different from one touching production. Acceptable deferred verification should scale inversely with the cost of error. Schwartz’s mistake wasn’t using ChatGPT for research — it was not verifying citations before submitting them to a federal judge.

Am I reviewing substance or presentation? If you read an AI-generated document and felt satisfied because it was well-organized and covered the right topics — that’s a presentation check. Presentation checks catch almost nothing that matters. Substance checks verify specific claims, confirm specific facts, ask whether specific commitments are accurate. These are different kinds of reading, and conflating them is how automation bias works in practice.

The habits that shift the default

The mechanism of automation bias is passive. You don’t decide to skip scrutiny — the output simply doesn’t trigger the scrutiny instinct. Countermeasures mean creating deliberate friction that doesn’t depend on you remembering to be skeptical.

Read AI output as if someone else wrote it. When you authored the draft, you fill in intent you had but didn’t write. Treating AI output as external removes that charity and forces you to read what’s actually there.

For code — implement the first instance of any unfamiliar pattern yourself. Write it before delegating repetitions. The first migration script is how you learn what migrations do. The tenth is mechanical. Delegation becomes safe once you understand what you’re delegating.

For summaries — go back to the source for anything that matters. The summary can’t show you what it left out. If the stakes are high enough that you needed a summary, they’re usually high enough to verify the key points against the original.

Match scrutiny to stakes, deliberately. Being skeptical of everything defeats the purpose of the tool. The useful habit isn’t general suspicion — it’s the specific pause to ask: what is this output for, and what happens if something in it is wrong?

The reframe

The Schwartz story spread through legal circles as a cautionary tale about trusting AI. That framing misses the point. AI tools are getting better, not worse. The output is more accurate, more nuanced, more useful with every model generation.

The point isn’t to trust AI less. It’s that the better AI gets, the more consequential it becomes to calibrate your trust correctly. When AI output was rough and obviously imperfect, the imperfections were visible. As it becomes polished and fluent and nearly always right, the cases where it’s wrong become harder to spot without deliberate effort — not easier.

Automation bias isn’t a problem that better models will solve. It’s a problem that better models will sharpen. The skill of calibrated engagement — knowing when to accept and when to look harder — becomes more valuable as the tool improves, not less.