We're Overestimating Medical AI — and Underestimating the Harm (Jessica Morley, Yale)

AI ethicist Jess Morley: these chatbots are giving medical advice — so regulate them as medical devices.

Part of The Agentic Patient, a Faces of Digital Health series on how patients actually use AI — which tools, which prompts, which safeguards. In this episode, host Tjaša Zajc sits down with Dr Jess Morley, Associate Research Scientist at the Yale Digital Ethics Center and a former AI subject-matter expert at the UK Department of Health and Social Care, for a clear-eyed account of where health AI is going wrong — and how to use it well anyway.

Morley argues we systematically overestimate what these tools can do and underestimate the harm. She makes the case for "skeptical optimism," explains why bioethics principles built for one-to-one care break down against many-to-many AI harms, and reframes ambient scribes as inference engines rather than transcription services — with real consequences for coding, billing and patient records. Then she gets practical: the guardrails, prompts and habits patients (and clinicians) can use today.

Guest: Dr Jessica Morley — Associate Research Scientist, Yale Digital Ethics Center; formerly UK Department of Health and Social Care and the Bennett Institute, University of Oxford.

What the conversation covers:
- Why "skeptically optimistic" is the honest position on health AI
- AI adoption as "a hammer looking for nails" — and what needs-led design would look like instead
- OpenEvidence, EU rules and the question of regulatory capture
- The DeepMind–Royal Free case and why law alone isn't enough
- Beneficence, non-maleficence, autonomy, justice — and where they fail for AI
- Ambient AI scribes, miscoding, billing inflation and phantom tests
- Paid vs free models and the widening access gap
- The "ask why" rule and knowing when to walk away from a chatbot
- Red-teaming your own assumptions and playing models off each other
- Building a personal "harness" with skills so AI works from your history
- The last-mile problem and the case for regulating LLMs as medical devices
- Whether AI is narrowing how clinicians think

Chapters:

02:50 — Intro: The Agentic Patient and the case for skeptical optimism
05:52 — "A hammer looking for nails": adoption pressure without a plan
07:25 — OpenEvidence, EU rules and regulatory capture
09:42 — The DeepMind–Royal Free lesson: why law needs ethics
13:29 — The bioethics principles and what they were built to do
19:40 — Autonomy, consent and the ambient-scribe problem
21:49 — Scribes as inference engines: miscoding, fraud and phantom tests
29:06 — Paid vs free models and the access gap
33:25 — Using AI safely: the "ask why" rule
37:38 — Knowing when to walk away: engagement design and degradation
44:58 — Red-teaming and playing models off each other
49:00 — Harnesses and skills: making the model work for you
51:38 — The last-mile problem and regulating AI as a medical device
58:00 — Does AI narrow the clinician's mind?

The Agentic Patient series: https://www.facesofdigitalhealth.com/agentic-patient-blog
Website: https://www.facesofdigitalhealth.com
Newsletter: https://fodh.substack.com
LinkedIn: https://www.linkedin.com/company/faces-of-digital-health