Considerations for the Responsible Use of Generative AI in Healthcare

Applying Guardrails to Generative AI Use Cases in Healthcare

Dr. Leith States, chief medical officer at the U.S. Department of Health and Human Services, described generative AI as a tool set that potentially could be applied to a variety of public health system challenges. However, one thing he said he finds concerning is a lack of governance in the U.S. While there’s been a movement to press forward with utilization, there hasn’t been an intentional approach to govern the technology, according to States.

In October 2023, the White House released its executive order on the use of AI. States said this document was highly prescriptive and lacked agility. However, HHS has adopted tenets from the order and created an AI task force and playbook.

“It’s starting to look like there’s coordination around a shared understanding of what it is we’re driving toward, and that’s been very refreshing,” he added.

Patricia MacTaggart, teaching instructor and program director at George Washington University, compared healthcare’s adoption of AI with that of electronic health records.

“We didn’t have a common understanding. We were in the same meeting talking about two different things,” she explained.

MacTaggart recommended creating a framework for discussions about AI to help the industry navigate conversations around implementation, whether the use is intended for patient engagement, clinical workflows or administrative efficiencies.

“All of those use cases need some guardrails, so we know the minimum bar but also the possibilities that we’re seeking and the realities of today,” she added. “This is innovation. There’s going to be some evolution. Some things are going to go right, and some aren’t. We need to understand those risks and apply each of those guardrails to each use case that we’re doing.”

EXPLORE: Three tips can help you master operational AI transformation.

The Importance of Data Quality and Transparency

“Data is the underlying foundation of successful generative AI,” said Coley, who explained that the more data included in the model, the more representative it is likely to be of the population at large. “It will provide you better models of prediction to train off of to make AI more effective and not a hindrance.”

MacTaggart pointed out that while healthcare organizations need good data for successful AI, they need a solid infrastructure to support data.

However, when it comes to generative AI tools, healthcare professionals may not have much information into the data set the model was trained on.

Heisey-Grove said the industry needs to start asking questions and demanding more transparency. She also suggested that organizations build their own knowledge bases and begin to provide their own data to large language models so that they can provide the right kinds of answers. While that won’t solve larger cultural issues or biases, it does help those engaging with the AI to know what data is involved and what data is missing. Heisey-Grove also recommended having a human verify the results.

“We can’t trust it on its own just yet,” she said.

Coley emphasized having the right people at the table to prevent bias and ensure representative data. If healthcare organizations are building models, she said, data scientists, health equity experts and a diverse group of people should be involved.

“We need people at the table who are always evaluating data for bias and bringing another lens,” she said.

States agreed, adding that it’s important to develop processes reflective of reality and not theory, which could be biased. Having the wrong people and the wrong data involved in a data model can lead to unreliable results.

As these tools get deployed across healthcare, MacTaggart said, it’s important to keep in mind that organizations should provide the right care at the right time with the right providers, modalities and appropriate use of algorithms.

Another reason healthcare organizations should be cautious about generative AI implementation is that not all healthcare professionals have the knowledge they need to engage with AI in a meaningful and responsible way. The industry needs to be realistic about how quickly it can implement these tools, MacTaggart said.

As public and population health professionals approach generative AI use, Heisey-Grove said, they should think of clinical data as one of many sources of data available within a community. It’s important to consider multiple types of data sources to create a more holistic picture of public and population health.

MacTaggart recommended that organizations start with administrative use cases that present less risk to patients. Having wins in that area can build trust as these organizations move toward use cases with more risk.

UP NEXT: An integrated data strategy supports healthcare AI success.