Healthcare's Alignment Problem: A Preexisting Condition

Translate

Healthcare's Alignment Problem: A Preexisting Condition

Or, Whose Agent Is It Anyway?

Robert Wachter

Oct 15, 2026

18

7

8

Share

The Hugging Face hack and Jacob Coxon’s post-Anthropic tweet have thrust AI alignment squarely into the middle of our public discourse.

The AI Alignment Problem, as seen by Gemini

In today’s AI safety debate, “alignment” refers to the gap between human values and the values (or at least the behavior) of an AI system. That kind of alignment certainly matters in healthcare. An AI without a moral compass that can access patient records and alter everything from medical histories to prescriptions would be awful. And as AI becomes embedded in ever more medical devices, you don’t need to be a fan of speculative fiction to imagine AI that kills by mucking around with pacemakers, defibrillators, insulin pumps, or surgical robots. Pretty scary stuff.

But healthcare’s bigger alignment problem isn’t about rogue machines. In fact, it’s a preexisting condition. The humans and institutions in healthcare have never been fully aligned with one another. Patients, clinicians, health systems, payers, manufacturers, and governments often want different things, and the rise of third-party payment has widened those gaps, making them far more consequential.

Economists have a name for this: the principal-agent problem. The person making a decision (the agent) doesn’t always share the goals of the person on whose behalf it’s made (the principal). For most of modern medicine’s history, the agent in question was the doctor. Increasingly, it will be an AI. As AI-enabled decision support becomes more robust, more persuasive, and perhaps even agentic, it will quietly encode one set of values and disseminate it at scale.

Basic idea of the Principal-Agent Problem (Wikipedia)

The key point is this: most clinical decisions don’t have a single correct answer. Instead, they have several defensible paths. Whether an AI decision-support tool recommends the “best” option (damn the costs) or the most cost-effective one (even if it isn’t optimal for an individual patient) is a values choice, not a technical one. If the machine is spring-loaded to optimize care for every individual, we’ve put our entire system, indeed our society, on a path toward bankruptcy. But make cost-effectiveness the default setting, and “AI death panels” will reenter the public discourse within a week.

The hope for AI in healthcare is that it can improve every element of healthcare’s value equation: quality, safety, access, convenience, equity in the numerator; critically, cost in the denominator. Cutting costs seems like it should be the easy part: less paperwork, fewer people in the billing department and the call center. Yet early evidence suggests that today’s AI is increasing costs, primarily by equipping providers with tools to “optimize” clinical documentation. (Payers, predictably, are responding in kind, doing their best to use AI to manage their costs, but for now, hospitals and doctors are winning this contest.)

This cat-and-mouse game will go on for a while, but it’s a sideshow. There’s an old saying in medicine: the most expensive technology in a hospital is the physician’s pen. The pen has given way to the keyboard, of course, but the principle holds. Physician pay is a modest slice of national health spending, while physicians’ decisions about what to test, what to prescribe, and whom to admit – and discharge from – the hospital drive most of the U.S.’s six trillion-dollar yearly healthcare spend. Replacing call-center staff with AI and giving doctors a few minutes back with AI scribes are worthwhile. But AI’s real impact on costs will come through its effect on clinical decisions.

When I’m on clinical service at UCSF Medical Center, I use OpenEvidence on my phone several times a day. It often makes testing and treatment recommendations, but it currently lives outside UCSF’s electronic health record, which means that it’s not teeing up orders for me. As these tools become embedded in the EHR and their recommendations become the default action, this issue will become increasingly important.

A 45-year-old man has had a week of back pain. He has no fever, no focal neurological symptoms, and no history of cancer or trauma. He’s tried Advil and rest, but the pain persists. Virtually every guideline, from the American College of Physicians to Choosing Wisely, advises against an MRI at this point. The scan is costly, and it is far more likely to turn up a false positive (an “incidentaloma“) than anything clinically meaningful. Yet an AI tuned to protect the doctor and health system from a malpractice suit might well recommend it. So might one that’s tuned, more quietly, to the health system’s bottom line: in a fee-for-service world, the MRI is a revenue-generating machine. Either way, it’s the old problem of overuse, now deliverable at scale.

A subtler but ultimately more important case is a test or treatment that truly helps, but at a profoundly unfavorable cost-benefit ratio. Imagine a new Alzheimer’s therapy that slows progression by several months and costs $250,000 a year. It seems like a foolish policy choice for a society to commit public dollars to pay for it, particularly in a nation with tens of millions of uninsured people and underfunded schools and infrastructure. But an AI tuned to recommend any therapy that might benefit the individual patient will recommend it every time. (Increasingly popular “right to try” laws might force systems to choose this setting for their AI.)

Now flip the incentives. A 58-year-old obese woman has pre-diabetes and early sleep apnea. Her doctor thinks a GLP-1 medication could change her trajectory. An AI tuned to the payer’s interests, or to a health system that has taken on financial risk through a capitated contract, might steer her toward “lifestyle modification first” and another recheck in six months, citing guidelines that are genuinely ambiguous. Nothing about this recommendation is wrong, exactly. But it’s also a choice, made on behalf of someone whose interests may differ from the patient’s, and it’s invisible. Multiply it by millions of decisions, and you have rationing that no one debated or voted on.

Health systems and clinicians have long grappled with benefit-cost tradeoffs. What’s different in the age of AI? In the pre-AI world, the physician generally chose the clinical strategy. The tension between institutional interests and the Hippocratic Oath was often present, and these tradeoffs were adjudicated via millions of individual decisions – sometimes reached only after wrenching debate. But, in a world in which AI tees up a recommendation that clinicians increasingly trust (a well-documented phenomenon called automation bias) and reflexively “click to accept,” the values of whoever built the tool are disseminated at scale. Someone else – an EHR vendor, a decision-support company, or a health system’s AI committee – is now choosing the values behind every clinical decision and doing so in a system that deflects scrutiny and bypasses debate. That’s a very big deal.

There’s another risk. Absent any transparency requirements, how can a patient or clinician know whether a recommendation was shaped by an undisclosed corporate relationship? It’s easy to imagine a well-placed donation by a device or drug manufacturer nudging AI-generated recommendations. Here too, corporate influence on clinical decisions is nothing new. But when companies court individual clinicians, strict reporting requirements now apply on both sides, and they have markedly toned down the old world of pharma-sponsored golf outings and lobster dinners. No comparable restrictions exist for AI companies or the firms that might influence them.

The newest party to this principal-agent problem is the patient himself. Tens of millions of people now check their symptoms and test results with ChatGPT Health or Claude for Healthcare before (or instead of) seeing a doctor. These tools are yet another form of decision support, now delivered to people who generally lack medical training.

Whose agent are they? It’s tempting to assume they work for the patient. But the companies that build them want them to be sticky, and a tool optimized for engagement isn’t the same as one optimized for a patient’s well-being. At a minimum, these tools are likely to be tuned to the individual in front of them, with little regard for cost (which, for an insured patient, is someone else’s problem).

Now, picture the office visit with a clinician. The patient arrives with an AI that says he needs the MRI. His doctor’s AI, tuned for cost-effectiveness and to prevent overuse, says he doesn’t. Who’s going to referee – and likely get beaten up by both sides? You guessed it: the clinician.

The stakes are too high for the underlying tradeoffs that may determine these tools’ output to remain opaque. But there’s a hopeful alternative. When these tradeoffs are encoded in software rather than scattered across millions of individual decisions, they can, for the first time, be made visible, debated, and audited. That’s how we avoid both implicit, inscrutable rationing and its equally harmful mirror image – unchecked overuse.

The most promising idea may also be the simplest. Rather than handing clinicians a single recommendation, decision-support tools could lay out the tradeoff: “If you’re optimizing for this patient’s individual benefit, choose Option A, though its cost-benefit ratio is unfavorable. If you’re seeking the most cost-effective choice that’s still within the standard of care, choose Option B.” Admittedly, a bit clunky, but it keeps clinicians and their patients as active participants in weighing trade-offs rather than passive acceptors of a value judgment made by a third party.

Beyond that, here are a few other ideas worth debating:

• Sunshine for AI. An Open Payments-style requirement that AI companies disclose any financial relationship that could influence clinical recommendations. This seems like an easy first step.

• Values labels. Should companies building clinical AI be required to disclose the values embedded in their tools, much as food makers disclose nutritional content? This’ll be hard to get right, but worth considering.

• Local governance. Should every health system have a multidisciplinary committee, including clinical content experts, informaticists, patient representatives, and ethicists, that reviews the cost-quality tradeoffs built into the decision support it deploys? Maybe.

Given that healthcare costs are bankrupting patients, as well as the governments and businesses that pay for healthcare, I hope these tools help us strike a reasonable balance between optimal care for individual patients and evidence-based judgments about cost-effectiveness. As these tradeoffs become explicit, we can expect a vigorous debate about rationing. And in the U.S., we can guarantee it will be politicized.

But failing to have that debate doesn’t avoid the choices. It just ensures that they’ll be built into clinical practice, at scale, without the public and professional scrutiny they deserve.

Disclosure: I am an advisor to Curai, an AI-enabled primary care company, serve on the board of The Doctors Company, a medical malpractice insurer, and have received a commitment of support from OpenEvidence for an upcoming documentary on AI and healthcare.