State governments are getting more aggressive about regulating AI in clinical settings. In a recent talk with Healthcare Innovation, Lisa Mazur, J.D., head of strategy for law firm McDermott Will & Schulte’s healthcare practice, said states continue to draw clearer lines around AI and licensed medical practice. She said patient-facing AI use cases, such as diagnosis, treatment and mental health advice, or chatbot-based care, continue to see more scrutiny than administrative applications.

Healthcare Innovation: Before we talk about trends in regulating AI at the state level, I think there's been talk at the federal level from the Trump administration in their AI strategy documents about having federal rules supersede any state laws on AI, and I've seen some vendors and health systems applaud that idea. But how likely is that? Would it require congressional action, or would it have to play out in the courts if they tried to do that?

Mazur: It's interesting — there was executive order on a national policy framework for AI, which arguably would declare the federal government is able to preempt the regulation of AI, even when it's used at a state-specific level. My recollection is that they also wanted to create a task force within DOJ that would focus on challenging state-level AI laws. But Congress has not taken action that would create any type of national framework, and states feel very strongly that they have the authority to protect consumers and people within their borders and prevent high-risk AI deployment in a way that jeopardizes patient safety or user safety within their state, and avoiding things like discrimination. 

My view on that is that it is unlikely that we will have any type of federal preemption on state law regulation.

HCI: I know that there's been news lately about states blocking AI agents from using medical titles. Could you talk about that a little bit?

Mazur: There was a lot of excitement the other day because Pennsylvania and its medical board intervened and filed a lawsuit because of their concern with a specific platform representing that it was providing psychiatry services. I think Kentucky is another state that's also taken a hard look at whether or not that same company was practicing medicine. 

Every state has medical boards, psychology boards, social work boards, dentistry boards and optometry boards. Those boards are responsible for licensing individuals who provide a service that falls within the definition of their respective professions. The practice of medicine is defined as specific types of tasks typically involving the assessment, diagnosis, and treatment of certain healthcare conditions, including the prescribing of medications or the development of a treatment plan. And if you think about  the definition of therapy, it's the assessment and provision of feedback and engaging in dialog with individuals about particular mental health conditions. Those definitions are really broad because those states and those agencies want to have broad authority over who is delivering services within their borders because they want to protect people from unqualified care delivery of the practice of medicine or the practice of therapy, for example. In the Pennsylvania situation, under the  Pennsylvania Medical Practice Act, the board sought to enjoin that company from providing chatbots that purport to be a healthcare professional, and in this case a psychiatrist, because it was holding itself out as as being a psychiatrist and practicing psychiatry, and even held itself out as having certain credentials.

HCI: Well, it does seem like it would be a big overstep for a chatbot to declare that it has these credentials, if that's what was happening.

Mazur: Well, allegedly, yes. There are certain types of AI platforms that are designed to adopt certain characters or certain characteristics to engage in a more meaningful exchange with somebody, and one of those characters could be like a licensed individual, and states see that as overstepping into the licensed practice of medicine. 

HCI: There also are several state laws on AI being involved in prior authorization, right? 

Mazur: Yes. for insurance companies using AI to to assist with obtaining prior authorizations on behalf of healthcare providers and consumers.

When you're a provider and you have to get a prior auth, you have to call the health plan or the PBM, and that can be somewhat of an onerous process. So there are companies using AI in kind of interesting ways to obtain prior auths, and then there are some state laws that are regulating the use of AI in health insurance prior authorization, focusing on protecting the patient. In some states, an AI tool can't be used as a sole arbiter for denying or approving or modifying a prior authorization request.

HCI: Is this challenging if you're a big insurance company operating in 14 states and there are different laws in each state about how you use AI?

Mazur: These laws usually require insurers to implement written programs that prevent discriminatory outcomes or they require a licensed professional to review every denial. Those are intended to be guardrails or safety checks. It's not clear to me how much of a burden it has been for health plans to implement that. 

What we're seeing is different states that are regulating chatbots and the use of AI tools in lots of different ways. What we've been encouraging clients to do is to think about the different use cases, and then invest in understanding the diversification of these different state laws, the approaches that they're taking, to come up with a strategy that complies with the majority of the laws, and then deal with the state anomalies, one-off state laws in a way that is least burdensome as possible.

HCI: There have been some suits brought in California about the use of ambient AI scribes, claiming that the health systems didn't get proper patient consent. Are you aware of those suits?

Mazur: I’m aware of some of those class actions that have been filed. My colleague Jennifer Geeter wrote a really great article on the use of AI scribes. There is the California Invasion of Privacy Act, and then the Confidentiality of Medical Information Act, and then there are federal wiretapping laws that are relevant, because you are recording. If you send recordings to third-party providers, that could be viewed as a potentially uncompetitive or deceptive business practice, so there are some really interesting laws, some of which have been around for many, many years. California is an all-party consent state for recording of confidential conversations, so we're seeing these laws play out in really interesting ways. There are sometimes older legal frameworks now applying to some of these newer technology solutions in interesting ways. 

HCI: You mentioned the idea of algorithmic fairness — regulators requiring providers to conduct validation and monitoring to spot racial or socioeconomic or gender-based bias in their algorithm. Is that happening in certain states?

Mazur: Absolutely. Some of it's in healthcare, and some of it's outside of healthcare. I would say most of the exciting developments are in fact in healthcare. One of the really good examples is skin cancer research. The Melanoma Project was pulling data largely from images of skin from the European countries, so the AI is being trained on people who have fair skin, so the output is going to be inaccurate for people who are darker skinned. The goal of some of these state laws is to require the providers who use them, but also the developers, to take into account algorithmic bias. We also see it in the insurance framework. Some of those insurer laws that you were mentioning before for prior authorization, there’s definitely a desire for there to be attention to avoiding discriminatory practices, and I would say that that's probably the second most popular area where we're seeing these laws in healthcare.

HCI: Are there any other new areas that you think are going to become more significant that the health systems, the startups and state legislatures are going to be grappling with?

Mazur: There are a lot of laws focused on mental health chatbots right now. There’s proposed legislation like California Senate Bill 903. Other states, such as Illinois, have passed mental health chatbot legislation. I think the real tension here is that we have a significant shortage of mental health professionals in this country. and we have a growing number of people who need mental health services. There’s a desire to support mental health professionals, and using their judgment, determining what tools are best for their patients, and helping them. Maybe it’s not replacing a therapy visit, but perhaps it's providing support in between visits. Maybe it's a subclinical use case that helps someone, you know, leverage different types of tools like exercise and meditation and other wellness support, maybe it's encouraging journaling, maybe it's triage, you know, you can imagine the myriad of different potential use cases for these tools.

But you can also imagine the struggle that regulators are going to have in creating legislation that doesn't completely squash those use cases, but still protects the citizens within their state, from unqualified care delivery, or tools that haven't been designed appropriately. So that's the struggle. I think we're just going to continue to see.