Artificial intelligence (AI) has moved from the fringes of healthcare innovation to the center of daily hospital operations. This sudden influx of unmanaged software — such as AI-assisted diagnostic imaging, ambient documentation and generative AI — has immense potential but is evolving at lightning speed and operating outside formal governance. 

IT teams need to be able to establish guardrails. Hospital technology leaders face a two-pronged challenge of not only capturing AI efficiencies but also preventing liabilities and vulnerabilities.

This article gives you actionable recommendations to establish AI governance and use the technology safely, effectively and responsibly. 

Operational Realities and Current Applications of Clinical AI

Departments are integrating AI capabilities across multiple clinical and administrative touchpoints to accelerate care delivery. Technology departments need a close-up view of workflows to understand what’s happening to data across environments. 

AI tools have become organically embedded in clinician workflows and platforms to streamline patient care. Today, radiology departments use automated image analysis to flag anomalies in X-rays, CT scans and MRIs. Pathology laboratories get digital assistance to optimize slide reviews, and emergency departments integrate risk-stratification models into triage systems to prioritize patient needs.

Ambient dictation solutions have introduced a significant operational layer by automating documentation and improving billing accuracy. We see this as AI-powered voice assistants listen to physician-patient conversations, generating real-time clinical notes that go straight to the electronic health record (EHR). Automating notes reduces documentation burdens to save time and lower burnout.

EHR software vendors have integrated AI to support clinical workflows. Built-in predictive logic supports real-time decision-making at the point of care with several helpful functionalities:

• Predictive alerts
• Medication reconciliation
• Patient deterioration scoring
• Care gap identification

As notes are sent to the EHR, AI continuously improves billing accuracy by scanning for items clinicians did but forgot to enter manually. These platforms can analyze specific properties and medications in patient records to recommend treatment options.

Hospital administrative staff are using AI to expedite tedious, time-consuming processes. Billing and coding departments use AI to assist with ICD code assignment and claims scrubbing, reducing turnaround times. Revenue cycle teams apply AI to predict claim denials and automate appeals, while HR leans on AI tools for recruiting and scheduling.

Administrative personnel rely on generative AI for daily operations and communications. Staffers turn to large language model (LLM)-based platforms to manage high-volume tasks:

• Draft emails
• Summarize meeting notes
• Write policy documents
• Respond to patient messages
• Look up clinical information

The Growing Challenge of Unsanctioned Shadow AI

With consumer AI tools readily available, healthcare organizations have countless technology systems to manage. The problem is that organizations lose control when workers log into personal accounts for quick answers.

Hospital personnel routinely adopt consumer AI tools — such as chatbots to research drug interactions and writing assistants to draft policy memos — that make their jobs easier. But they don’t fully appreciate the AI governance responsibility involved and fail to request IT approval, creating an invisible risk architecture.

Tools that capture sensitive patient information compromise privacy and compliance. When employees input protected health information (PHI), such as a patient's name, date of birth, diagnosis, and medication lists into consumer software, it could be used to train public LLMs stored on servers outside the Business Associate Agreement (BAA) and transmitted to jurisdictions with different data protection standards. This constitutes a breach of federal privacy laws, and federal enforcement agencies will hold healthcare facilities accountable.

Even as the rest of the industry works to create guardrails for AI governance, the Health Insurance Portability and Accountability Act (HIPAA) doesn’t carve out exceptions. All systems that receive, process, transmit or store PHI, including AI tools, are subject to HIPAA Privacy and Security Rules. Additionally, if an employee enters patient data into a commercial platform, the hospital could face consequences from the Office for Civil Rights, which holds covered entities responsible for workforce actions in the event of compliance audits or investigations.

Unregulated third-party or “shadow AI” adoption puts up multiple roadblocks as hospital IT teams work to protect their organizations from external liabilities. When staff members bypass established channels, it compromises the security perimeter.

1. Visibility: IT has no reliable way to monitor AI usage and organizational data sharing when they lack appropriate tracking tools and staffers bypass their approval.
2. Accountability: If a data breach, HIPAA violation or patient safety event occurs based on an AI recommendation, IT can’t investigate or remediate it if the team doesn’t know it exists.
3. Culture: Employees work around governance if they believe IT will always say "no" to useful tools.

Confidential data has never been more vulnerable. A 2026 Cyberhaven AI report found that nearly 40% of employee interactions with generative AI across industries such as healthcare involved sharing sensitive data. A substantial amount of this data is shared through personal accounts on popular platforms (ChatGPT, Gemini, Claude and Perplexity), further obscuring organizational visibility and control. IT needs to be part of each implementation to ensure safety.

Regulatory Compliance and Clinical Liability Challenges

Regardless of its operational and efficiency benefits, healthcare AI raises legal, ethical and clinical concerns traditional risk management can’t answer. Technology leaders bear the weight of ensuring that automation doesn’t compromise patient care or organizational safety.

Relying on AI for clinical decision-making may create efficiencies, but it could also undercut expertise and harm patients. If a model suggests an incorrect drug dosage, misclassifies imaging, or generates a factually incorrect clinical recommendation, the consequences could be life-altering. 

That’s why the Food and Drug Administration (FDA) regulates AI tools that qualify as software as a medical device and are used for diagnosing, treating or preventing disease. Hospitals that use tools from external providers must manage them as traceable supply partners, use only FDA-approved versions, follow manufacturer instructions, and create processes for monitoring and reporting adverse events. 

Healthcare doesn’t yet have industry-wide AI governance standards. When AI-enabled software contributes to negligent outcomes, the liability rests with you, not the vendor. IT and legal teams should scrutinize vendor contracts to understand their responsibilities, assessing indemnification clauses, data use agreements, limitations of liability, and the scope of the vendor's BAA to cover PHI. 

AI systems trained on narrow historical datasets notoriously amplify biases and health disparities among diverse patient populations. No matter how well an algorithm performs generally, it may falter in its accuracy when handling data for groups such as:

• Elderly patients
• Patients of color
• Female patients
• Patients with multiple comorbidities

Hospital IT, informatics and compliance teams must evaluate AI tools for bias across care environments before deployment and continuously over time to ensure consistent performance across groups.

Actionable Guidance for Building a Responsible Governance Framework

Create a multilayered framework to protect your data while supporting clinical innovation. 

Constant interaction with automation can actually drive cognitive fatigue, so proven processes and collaboration across the healthcare continuum are necessary.

• Deploy internal oversight dashboards to track token consumption and inefficient prompt patterns.
• Monitor traffic to AI services to understand work and personal use patterns.
• Integrate data from management platforms to triangulate AI usage with delivery velocity and forecasting.

Enforce a mandatory registration protocol documenting every active intelligence application, underlying data permission, and verified compliance status. IT teams should conduct regular audits of all AI tools across the organization to discover unauthorized platform connections before data exposure occurs.

• Review software procurement records.
• Examine network traffic logs to known AI platforms.
• Capture intended use, data access, and regulatory classification of each tool. 

The register should be a living resource. Whenever new tools are requested or discovered, update it immediately. 

Establishing rigorous technical boundaries can help to control how data interacts with external software services. Implement advanced tools to prevent unmonitored data transmission. 

Organizations can create a firewall against data exposure to monitor, control access to, and even block the transmission of sensitive records to unapproved AI services. Combine this with tiered controls to block high-risk consumer AI tools while also giving employees a transparent evaluation pathway to formally request access to AI-enabled software. 

Form an oversight group of informatics leaders, compliance officers, legal counsel, security experts and end users to continually evaluate and approve new tools before deployment. Document approval decisions and conditions of use based on the assessment of each tool’s:

• Security architecture and vulnerability surface
• Data handling and vendor data use policies
• HIPAA compliance and BAA readiness
• FDA regulatory status
• Clinical validation and evidence base
• Bias assessments
• Organizational strategy alignment

Once a tool is in place, ensure it's serving the intended purpose(s). Monitor them on an ongoing basis for performance, safety signals and compliance:

• Review audit logs
• Establish reporting mechanisms to flag concerns
• Conduct periodic reviews of performance data

For clinical AI tools, IT should work with clinical informatics and quality teams to define key performance indicators and track vendor updates to ensure they can adapt to changes to how the tool behaves.

Publish a clear, plain-language AI use policy that explicitly defines the organization’s parameters for integrating AI into workflows. It should cover several key considerations:

• AI tools approved for specific use cases
• Rules regarding submitting PHI to unapproved platforms
• Processes for requesting approval of new tools
• Expectations around verifying AI-generated outputs
• Reporting requirements for concerns related to AI outputs 

Make sure everyone in the organization understands the policy by communicating it broadly and reinforcing the particulars through onboarding, training, department meetings and ongoing AI approvals.

Manage AI Responsibly to Leverage Its Benefits

Hospital technology leaders face increasing demands to integrate AI solutions but must be diligent about implementing them to protect their organization’s patients and administrative operations. To truly reduce risk, the industry needs to create governance measures and review processes, and it takes all of us.

Haven't made the leap yet? Your organization can reap the efficiencies of AI safely and securely with proper oversight and governance. Implemented responsibly, technology of this magnitude can bring benefits that far outweigh any oversight costs.

Chris Mercier, vice president of research and development at Juno Health, has extensive experience that includes more than 30 years of IT experience, with 25 years focused on healthcare IT and customer service excellence. Throughout his career, he has designed and implemented multiple healthcare billing and electronic health record applications while overseeing complex system integrations and migrations.