By Harvey Castro, MD, MBA, Practicing Emergency Physician, Healthcare AI Advisor (DR GPT)
LinkedIn: Harvey Castro, MD, MBA
Most hospital AI projects look convincing in a conference room. The slide deck is clean. The pilot cohort is curated. The demo patient has a complete chart. Then the model meets a real emergency department at 2 a.m., and the story changes.
I still work that shift. The chart is thin. The patient is undifferentiated. Seconds decide outcomes. That gap between demo and workflow is not a mystery of algorithms. It is a failure of how we choose, govern, and implement clinical AI for the people who actually use it: physicians, nurses, and care teams under pressure.
Three reasons demos succeed and floors fail
First, demos optimize for accuracy on clean data. Real care optimizes for time, trust, and action under incomplete information. A model that ranks well on a retrospective dataset can still slow a clinician if it takes six clicks, opens in a separate window, or floods the inbox with alerts that do not change what happens next.
Second, pilots often exclude the hard cases. Rural sites, night shifts, boarding, language barriers, and sparse documentation are where value-based care and quality reporting actually break. If your AI never tests on those conditions, do not be surprised when adoption stalls outside the flagship hospital.
Third, governance arrives late. Procurement buys a tool. IT integrates a feed. Clinical leadership meets after go-live. By then the product has already trained staff to ignore it. Physician-led governance has to start before the contract, not after the outage.
A practical 2 a.m. test for clinical AI
Before another AI purchase, ask one question in the language of operations, not marketing: if this fails in a crowded ER at 2 a.m., does it belong on the floor? That test forces vendor-neutral criteria your CIO, CMO, CMIO, and quality leaders can share.
Does it reduce cognitive load for the frontline clinician, or add another screen to reconcile? Does it work when the chart is incomplete? Is the recommendation explainable enough that a physician can defend it in peer review? Can nursing and pharmacy act on the same signal without a separate workflow? Is there a clear owner when the model drifts, and a clear path to turn it off?
Those questions map to what Health IT leaders already manage for EHR adoption, HIE, interoperability, and analytics. Guidance from the Office of the National Coordinator on artificial intelligence in health care and the FDA framework for AI and machine learning in software as a medical device both point to the same operational truth: AI is not a separate category of magic. It is another clinical information system that must survive the same stress tests as the rest of the stack.
Physician-led governance that changes outcomes
The National Academy of Medicine report on AI in health care warned about the gap between hope and peril years ago. Hospitals that keep AI alive after the pilot do a few unglamorous things well.
They put practicing clinicians in the selection room with equal weight to IT and finance. They define success as workflow minutes saved, alert burden reduced, or quality-measure movement, not model AUC alone. They require prospective monitoring after go-live, with thresholds that trigger pause or rollback. They train for override literacy: when to trust the suggestion, when to discard it, and how to document either choice.
They also align AI with federal and payer reality. CMS value-based care priorities, quality reporting, and rural access constraints do not care how impressive the demo was. If a tool cannot support safer, faster decisions that improve measurable care, it is entertainment with a software license.
What to do this quarter
Inventory every clinical AI tool already live. Kill or pause anything staff routinely ignore. For new buys, write the 2 a.m. test into the RFP and the governance charter. Require a physician owner, a nurse owner, and an IT owner before signature. Measure alert volume and time to action in the first 90 days. Publish those numbers to the medical executive committee the same way you would any other clinical risk.
Hospital AI will keep failing the floor until we stop buying demos and start governing workflows. The technology is ready enough. The question is whether leadership will let the night shift veto what the conference room applauds.