Health system leaders aren’t short on questions about artificial intelligence. Boards want strategy. Clinicians want safety and workflow integration. Regulators want accountability. But beneath all of that sits a more fundamental issue: what, exactly, are clinical AI systems optimized to do, and who benefits?

That question anchored a recent New England Journal of Medicine (NEJM) AI virtual event on value alignment and incentive divergence in clinical AI. Across payers, providers and technology companies, panelists returned to the same theme: the hardest part of AI in healthcare isn’t building the models, it’s aligning them with the realities of care delivery.

Global Models, Local Realities

AI may be built at scale, but it doesn’t perform that way automatically. David Rhew, M.D., global chief medical officer at Microsoft, emphasized that while evidence-based medicine provides a shared foundation, implementation is inherently local. Models trained and validated in one environment don’t reliably translate to another without adjustment.

His core point was that healthcare organizations can’t assume AI will generalize cleanly. It must be continuously evaluated and adapted to local populations, workflows and cultural expectations, or risk underperforming in ways that aren’t immediately obvious.

That challenge is driving more collaborative governance approaches. Rhew pointed to growing efforts among health systems to share insights about AI performance and oversight, creating early versions of what amounts to a learning network for AI deployment.

Scaling AI Requires Guardrails and Accountability

For large organizations, alignment challenges multiply quickly. At Elevance Health, which   more than 100 million individuals across diverse populations and payer models, AI deployment requires what Ashok Chennuru, chief data and digital AI transformation officer, Elevance Health, described as a rigorous, multi-dimensional governance framework.

That includes evaluating bias, fairness, explainability, privacy and robustness, but also looking beyond internal systems. AI decisions, he noted, sit within a broader ecosystem that includes providers, regulators and life sciences companies.

One of the clearest real-world examples is prior authorization. Elevance is using AI to automate portions of the authorization process, but with a critical safeguard: human review remains in place for denials. The organization also continuously audits AI-driven decisions to ensure they hold up under regulatory scrutiny and can be clearly explained.

Just as important, Chennuru emphasized an affordability lens, using integrated data — from claims to clinical information — to guide decisions that balance cost and care quality.

The goal is straightforward, even if execution isn’t: keep the patient at the center while navigating complex financial and regulatory pressures.

Data quality and context still define outcomes

While governance frameworks are evolving, some fundamentals haven’t changed. Meera Kataria Atkins, M.D., chief medical officer at Lyric, brought the conversation back to a familiar but often underestimated issue: data quality and context drive everything.

Her message was simple: AI systems are only as good as the data and assumptions behind them. And in healthcare, context can dramatically change what “good” looks like. Claims data, for example, offers valuable retrospective insight, but it’s incomplete on its own. Clinical data, real-world constraints, and access issues all shape outcomes in ways models must account for.

She pointed to a practical example: even when evidence-based guidelines recommend a specific therapy, that recommendation breaks down if patients can’t access the treatment. In those cases, an “optimal” AI recommendation may not be useful, or even appropriate.

Atkins also challenged AI developers to do more than reinforce existing thinking. Instead, she argued, models should help clinicians identify blind spots and missing considerations, especially in high-stakes decisions like patient safety.

Trust is the True Rate Limiter

If there was one word that surfaced repeatedly throughout the conversation, it was trust. Andrew Ibrahim, M.D., chief clinical officer at Viz.ai and a practicing surgeon, put it bluntly: AI adoption will move only as fast as clinicians trust it.

That trust depends heavily on transparency. Clinicians want to understand how a recommendation was generated, including what data was used, what signals were prioritized and how conclusions were reached. Black-box systems, he suggested, simply won’t gain traction in clinical environments.

Just as telling is how quickly trust can erode. Ibrahim noted that even small drops in clinician engagement can signal deeper confidence issues and require immediate attention. At the same time, the inputs into clinical decision-making are expanding. Patients are increasingly bringing their own AI-generated insights into encounters, creating a more complex, multi-source information environment.

Rather than resisting that trend, Ibrahim sees it as inevitable. The future, he suggested, is multimodal — integrating clinical data, imaging, patient-generated insights and more into a unified decision framework. Notably, he pushed back on the idea that technology itself is the primary constraint. In his view, the bigger barriers are governance, policy and financial structures that haven’t yet caught up to the pace of innovation.

Beyond Efficiency: AI’s larger opportunity

While much of the conversation focused on risks and constraints, panelists also pointed to a larger opportunity. Rhew described emerging approaches that break complex clinical reasoning into multiple coordinated AI components, such as diagnosis, testing, evidence synthesis and cost evaluation, rather than relying on a single model.

This modular approach not only improves performance, but also makes systems easier to monitor and adjust. If one component underperforms, it can be refined without overhauling the entire system.

More broadly, he argued that healthcare organizations may be underestimating AI’s potential by focusing too narrowly on efficiency. Yes, AI can streamline workflows and reduce administrative burden. But its more transformative potential lies in enabling new kinds of work, such as solving problems at a scale or complexity that wasn’t previously possible. That shift, from incremental improvement to fundamental redesign, is where many believe AI’s true impact will emerge.

Alignment is the Real Differentiator

Across the discussion, one conclusion stood out: clinical AI success won’t be determined by model performance alone. Instead, it will depend on how well organizations align:

• incentives across stakeholders,
• data strategies across systems,
• governance with real-world use, and
• technology with patient-centered outcomes.

The technology is advancing rapidly. But without that alignment, even the most sophisticated tools risk falling short. For healthcare leaders, the mandate is becoming clearer, which is to focus less on what AI can do, and more on what it is designed, incentivized and trusted to do.