Tripathi: The primary responsibility was very similar to what I'm doing today here at Mayo, which is about reviewing, vetting, and approving the uses of AI within the department itself. That’s a simple statement, but then you think about how big that is — CMS, NIH, Indian Health Service, etc. We ended up getting involved with the VA because the VA had some solutions that were crossing over into Indian Health Service. I had to set up a process within our office to say, how do we do a broad scan of the AI solutions that are in place across the department? What is the framework that we're going to use to review those in pretty rapid cycle? It was very pressing then, too, because a number of these things were already in use, and the executive order that came out said that if they're not reviewed before the end of the calendar year, you have to stop using them. We talked to the head of the IHS who said, wait a minute, we've got five of these things used in clinical practice. We can't shut these things down. Anyway, there was a lot of urgency to set up a framework that is functionally similar to what I'm doing here at Mayo.

HCI: When you first came to Mayo, were there AI governance processes already in place?

Tripathi: There were. I would describe it as a very decentralized governance process. All the pieces were there, but the challenge that we faced was that Mayo has always been a very innovation-driven culture, going all the way back to Will and Charlie Mayo, who talked about bringing together different parts of Mayo in innovative ways to develop solutions. 

With the advent of AI technologies, where you have low-code and no-code capabilities to create something pretty powerful if you have a good idea, what we started to see and we're still seeing is a burgeoning of solutions coming up from the bottom. Mayo’s culture is basically unbridled innovation. [CEO] Dr. Gianrico Farrugia has pounded the table saying people should be using agents, they should be using AI. Everyone needs to understand this technology and use it in ways that they think are going to benefit our patients. We have roughly 450 solutions in our pipeline right now and 128 that are used in clinical practice. What we found was that our governance process hadn’t adjusted to that increase in volume and complexity of solutions. 

What was happening is solutions would come up and need to get a security review, and they needed to get a privacy review, and they needed to get a technical integration review. But those would go to different parts of the enterprise, and of course, when you have that many of these, some of them start getting lost or slowed down because there are just so many of them. Also, they are raising issues that are unique to AI that might fall in the cracks between departments. People then stop and say, "Okay, wait a minute. We need to figure it out,” and that was slowing things down even further. 

So when I started talking to Dr. Farrugia about coming here, he asked if I was interested in helping develop a governance process that could not only be a filter and a reviewer, but an enabler, an accelerator to tap into that incredible innovation and get it into the hands of clinicians safely.

HCI: Does that involve setting up committee structures where people who work in the field participate on committees to decide these things, or is it more of a dedicated team that's doing the reviewing, vetting and approving work full time?

Tripathi: A little bit of both, but what we tried to do is get rid of a lot of committee structures that were very ad hoc. We believe at Mayo that this is a transformative moment for medicine, and AI and agentic solutions are a key part of that. We see this as strategically important to the delivery of high-quality care to our patients, and if it's something that's strategically important, we need to have an executive leader who is directly responsible for it, and that's what we didn't have.

Prior to my coming, there was the head of risk, the head of the IRB, etc. They are great people, and they're doing their jobs. But if this is something strategically important, you need an AI exec who cuts across all of those to make sure the solutions are getting the right attention, have the right policies in place, and then move them forward. I looked at the various things that are important from a review and performance perspective. We need to think about privacy. We need to think about security. We worked with our different colleagues who are experts in those areas, and then we created a unified set of policies and a framework for how we would evaluate these. 

I have a department and a dedicated team. We vet every solution that comes through. We said we're going to do that in a very high-velocity, AI-enabled manner to make sure that we're getting the kind of reviews that are necessary across all these domains, but that there is a point of accountability to see that through to the end. 

I report directly to Dr. Farrugia, which both he and I believe is incredibly important because it allows me to speak for the enterprise, and it also allows me to balance all the different considerations as we think about what's an important solution. 

We need to think about what's most important for our patients, and what's the benefit that we're going to get, and then how do we think about risk mitigation along the path for all of these and then make an executive-level decision on that. That's what our previous structure, because it was so decentralized, didn't have, and that was the importance of having an executive level reporting directly to the CEO to be able to make those decisions.

HCI: Is there a focus on measuring return on investment or the impact on clinical outcomes? Would that take place at your level or at a clinical department level?

Tripathi: In keeping with that very decentralized bottom-up innovation strategy, we also believe very strongly that the product proponents who are developing that product are the ones who have to maintain the responsibility for the lifecycle of that product. We establish a set of policies through my department where we cover the intended use of the product: Where do you intend to use it? What are the purposes that you're using it for? What are the privacy security, performance, patient safety considerations? How will post-deployment monitoring happen? 

We say you are required to do performance evaluation for clinical benefit and here are the parameters for doing that. What's an industry benchmark? What's the current standard of care? How will we measure the current standard of care? But we put that on the proponent and say it's up to you to do this evaluation and then present it to us within these parameters. Then we will evaluate that and we'll talk to you about whether we think that is a valid methodology and a valid approach. 

We allow them to do the analysis, but then we review it to make sure that it's robust and it's in keeping with Mayo requirements.

HCI: Is there a different or separate process for vetting and monitoring third-party tools vs. things that are developed in-house?

Tripathi: There isn't. At the end of the day, we cover vended products, co-developed products, and internally developed products. But if it's a vended product, for example, it'll come into the pipeline here through our third-party risk management group, which does all of the vetting of third-party solutions. They are looking at things like financial stability and the business associate agreement. If it’s an AI solution, then that comes to us for the AI portion and we do the same work with them. It’s a little bit different with a vended product because they're the ones who are the legal manufacturer, and ultimately they're responsible for the performance of the product, and that's a part of our contractual relationship with them. But we do require that they submit a set of evaluations for the product, so that we're able to assess it. If they're not producing the kind of evaluations that we think we need, we'll build more mitigations on our side. That means we are going to have to put additional protections on this product until we have our own data. We’re not going to take anyone's word for it if they say “Trust us. it's fine, but we can't share the data with you.”

HCI: We’ve written about a couple of new startup companies in post-deployment monitoring. One of them's a Duke Health spinout called Vega Health. They provide a platform to help health systems with continuous post-deployment monitoring of validated clinical AI tools. I'm wondering if Mayo has something like that or sees a need for something like that. You mentioned 128 AI tools already in action. Do you need a platform to be able to keep track of all those?

Tripathi: We don't have either a vended platform or a singular platform at Mayo. It’s federated. We treat it the same way that I was just describing, which is we have policies that require that product proponents have post-deployment monitoring according to a set of requirements, and it's really up to them to develop the ability to do that post-deployment monitoring. Those are things that we continue to put in place, and we’re building more and more infrastructure for that to be more and more automated. 

I know there's Vega and a number of other companies like that. But in an organization that is as large and complex as Mayo with multiple geographic sites and very complex architecture, driving every solution through a singular platform is really challenging. It's a lot of additional work. From a policy perspective, we say, "Look, we're not going to tell you which platform you have to use, or that you have to build a whole set of APIs directly to this, but we are going to say that you are required to do the monitoring according to these requirements.” We’ll see. Maybe over time it'll make sense to have a singular platform, but I think that it makes it very complex. 

HCI: If we look across the the country at AI governance at all kinds of health systems, do you see more similarities or differences? And is there a role for HHS or groups like CHAI or the Joint Commission to develop best practices or frameworks? 

The last point I'll make involves smaller systems. As I was describing, we have internally developed, co-developed, and vended products. With an internally developed solution, Mayo is responsible for everything, right? We're the manufacturer. So, from an FDA perspective, if it is covered by regulation, or even product liability or whatever it is, we're responsible as a manufacturer. But then we're responsible on the other end as the user.

With a product, we're not the legal manufacturer, but we are responsible as the user. So it's a different kind of governance consideration that you've set up. Organizations that are smaller and/or have fewer resources may have a different strategic perspective. If the majority of what you're doing is with a vendor through Epic or Oracle or eClinicalWorks, that's a very different kind of governance approach that you would take. It's much more contractual, pushing liability as far as you can to the other party, all of that. 

For us, something like 75% of our solutions are internally developed. So if you look across organizations, you would see a difference based on the composition of their pipeline.

HCI: Could you talk about how the work you're doing fits in with the Mayo Clinic Platform and the work that Dr. John Halamka and his colleagues are doing there?

Tripathi: We’re complementary. The Mayo Clinic platform is primarily externally facing, with the idea being that we want to be able to leverage and disseminate as much as possible, and get participation in a platform, technical, and business model for the ability to use shared infrastructure and shared data for multiple parties to be able to build innovative technologies and make it available to as many people as we possibly can outside of Mayo Clinic. It’s about it being an innovation platform and a collaborative opportunity, bringing to bear Mayo expertise, Mayo data assets, but also the expertise and data assets of vendors as well as other providers, but primarily with an external focus.

My work is about Mayo Clinic’s clinical practice itself and which solutions we're going to use in the clinical practice. To the extent that there are some solutions that come in through the Mayo Clinic platform that we want to use in the practice, then they would come through us as another product for us to review. From my perch, I view the Mayo Clinic Platform as being another pipeline of the many pipelines that are feeding the Mayo Clinic practice.

HCI: Recently I was watching a presentation by an Anthropic cybersecurity exec —  kind of a scary talk about how AI agents can now start implementing complex cyber attacks. Is that a topic you have to worry about or can you leave that to the CISO and CIO?

Tripathi: I was going to say that fortunately our CISO deals with that. But no, we do have to get involved in that somewhat. Now the primary responsibility, obviously, is with our our CIO and CISO team. However, this does touch on our policy requirements. For example, we have a host of patient-facing agentic solutions that we’re doing a lot of work on. We want to make those available through a patient experience platform that we think is going to be really innovative. But for any public-facing user interface that's got AI in it, you’ve got to have a special set of considerations related to privacy protection and security protection to protect against, for example, things that are called in the industry prompt injection.

Could a bad actor come in and start putting in a whole bunch of stuff in the prompts that all of a sudden the LLM starts doing wacko things? It could just give crazy answers that would not be good from a brand perspective or a patient relationship perspective. But maybe it's harmless. Or it could do very harmful things. It could convince it to give you data from other patients, for example. They could jailbreak it. There's data poisoning that you worry about. So we specifically have considerations in our policies that deal with the risk calibration or the risk tiering of those. And to the extent you’re working with an external partner, we have to have an additional set of security considerations. Of if you are exposing an API or some type of interface in a public manner to users outside of the Mayo ecosystem, that’s our highest level, where now all of a sudden we have to have all the security considerations in place.

HCI: Finally, what are you finding the most rewarding and most challenging about this job so far? 

Tripathi: Well, the rewarding part is easy. It is an amazing privilege to be here. Mayo truly believes that these technologies are transformative, and we've got leadership from the board and the CEO who are fully embracing that to improve patient outcomes. I was the chief AI officer at HHS. But when I got here 15 months ago, I had no idea how much was going on in a place like Mayo. I remember coming to Mayo and saying, "Whoa, we're building our own multimodal foundation models? I had no idea that we were doing that. I thought everyone was just using the the large language models from  OpenAI.” That was amazing to see. 

Certainly one of the challenging things is how do we keep pace with the technology in a way that allows us to maintain the ability to get these solutions in the hands of our clinicians, but in a manner that we know that they're high-performing, to deliver a benefit that a patient's going to experience in a way that is safe and trustworthy and reliable. Every new agentic product that comes to our door tests some of these policy frameworks. All of a sudden they're talking about doing something that we haven't really thought about, so let's rapid-cycle figure out what that new policy has to be. Let me consult our security experts, our private privacy experts. Now let's get that policy in place and then help them move forward. I think that's the ongoing challenge. 

The other one that a lot of the academic medical center are facing as well is that there are a lot of solutions now coming, especially if you're on Epic, for example. Epic has a lot of solutions coming in and we're an adopter of those. And then you have your own torrential river of internal development. Now you start to have the problem of whether some of these are redundant. Which one do I want to choose? And they don't always come at the same time. One comes six months later, and now I've got two solutions, and actually the one that came later could be better than the one I already implemented, but that one's already implemented. So now what do I do? I don't want to put two out there because of cost, but also confusion. Our providers are already overburdened, and these technologies are supposed to reduce the burden on them, not increase the burden. 

HCI: Is there a role there for the chief medical information officers and/or the CIO to make a call on that? 

Tripathi: I'm a part of the clinical practice leadership, and that's where we believe those decisions have to be made. You mentioned the CIO. We have a great CIO. But frankly those decisions, I would argue, need to be made by the clinical leadership. In a regular business, it would be the business owner, not the technology people, making the call. The business owner needs to say, this is the solution that's going to provide the best clinical benefit for us. I'm working with the clinical practice leadership now to have an intensive focus on how we weigh these products against each other. 

Tripathi recently appeared on the Tomorrow's Cure podcast with other Mayo Clinic leaders to talk about AI’s impact.