From a Patient Perspective, Governance Is Crucial for Ambient Listening Tools

HEALTHTECH: From the patient perspective, how would you describe the state of best practices today?

PRATT: We are in a development stage of what AI is and where AI governance is going, and that will influence our healthcare systems and patients in ways that haven’t been fully thought about. This may be the first time in a long time that patients will have a voice in what is happening with their healthcare. With ambient AI, now is the time to set the rules — not to wait until after the fact but to minimize what can go wrong ahead of time.

CRUMBLY: For someone going through a traumatic medical issue, seeing a specific human is very important. When you put AI on top of that, there are questions: Is the proper information recorded? Who will it be shared with? If something was missed, why wasn’t it properly documented? Is the AI accurately capturing and interpreting? These are all concerns that need to be thought through.

CHARLES: One of my biggest concerns is that the majority of communication between a patient and their clinician is nonverbal: Patients shrug, look depressed, shake their head, cry. A clinician uses intuition to determine that a patient is withholding something. None of that can be captured by a listening device. How do we capture the full patient encounter in a way that is truthful and consistent with the communication that actually occurred?

PRATT: Currently, ambient AI use isn’t consistent across hospitals. There’s a lack of standardization around accuracy, patients’ ability to reference what’s been documented, policies about data retention or how clinicians share that they’re using ambient AI. I’ve been to multiple locations, and every single person tells you differently whether they’re using it.

CHARLES: Most organizations introduce ambient AI slowly, often starting with volunteer clinicians in one discipline at a time. What’s a continuous challenge is that organizations are learning by doing without well-established frameworks for how to implement it. It works best when organizations pursue this slowly and gradually, learning from others and from their experiences about what makes sense for their values and mission. 

HEALTHTECH: What are some of the risks associated with a lack of governance?

PRATT: Data security is a big part of this. When facilities are collecting data without governance in place, what is collected, how it’s collected and who has access become critical issues. There is now a recorded patient voice at a time when hacking and cybercrime are occurring frequently, and several hospitals have already had data stolen. How are we going to protect patients’ data, and who is responsible?

CHARLES: The corollary of security is privacy. Patients share things with clinicians that they don’t share with anybody else — information that is private and potentially stigmatizing. There’s already a security risk related to third-party vendors managing audio recordings and transcription, but even internal quality assurance teams may listen to recordings to make sure they’re accurate. That’s not something patients would necessarily expect, that people outside the care team would listen to the recording, so that feels like a violation of privacy.

Patient access also comes up. Do patients have a right to their recordings and transcriptions? Organizations are determining how long to save recordings. Could attorneys access them in a malpractice lawsuit? Could law enforcement access them under a court order? These are huge issues that arise from the nature of the recording itself. 

Another risk is AI automation complacency. We know that clinicians are rushed and often tired, and there is a risk that they could accept a transcription without editing it enough and that incorrect information could become part of the patient’s medical record.

CRUMBLY: Particularly in the African-American community, researchers and advocates are looking at the European Union model to ask what rights people have to their data. Tuskegee University alums such as James Felton Keith have written about advocating for not only patients’ rights but also broader data rights for Americans and people around the globe. That’s why governance is so important, particularly in the U.S., as AI is growing so quickly.

GET EXPERT HELP: Transform patient care with data and artificial intelligence.

HEALTHTECH: What best practices for patient consent and notification do you recommend?

CHARLES: The basic premise is that patients should be treated as autonomous individuals who have a choice about how and what to communicate and the right to receive information needed to make an informed decision. Patients need clear notice, meaningful choice and the opportunity to receive written information before recording begins.

PRATT: Ambient AI is a layered process, and consent practices should recognize that. When a patient says, “I want care,” it may not mean, “I also want you to record information or use ambient AI for other purposes.” A best practice is to separate those aspects — patients should not feel that one requires the other. Plain language is critical. Saying, “All of your information is being transcribed by AI” doesn’t tell patients anything. Does recording start when they walk in the door? When the physician comes in? What about “smart” rooms with recording capabilities built in? Is it still recording when the physician leaves? Organizations that are doing this well are creating a layered, continual process with explicit information about what’s being recorded, how and what patients’ rights are regarding access, and corrections.

Click the banner below to sign up for HealthTech’s weekly newsletter.

HEALTHTECH: Looking ahead, what would you like to see happen in healthcare around ambient AI?

CHARLES: Organizations need to tighten their policies around recording retention and data access and be very clear about patient expectations. Digital health technologies are not an IT issue; they are a business issue. Organizations cannot view ambient AI scribes as solely an IT component and simply trust the IT department. Governance needs representation from a diverse body, and patient advocates are a critical voice that needs to be there.

CRUMBLY: I see this technology as the tip of the iceberg. As we have more listening devices, they will raise more questions about data. From a diversity perspective, there’s a question about trust with data sharing. If you remember Henrietta Lacks — a patient whose cancer cells were taken in 1951 without her consent and used for research — that trust is still a challenge and a major issue here. It’s important for healthcare entities to have processes that show people, step by step, what occurred and how their data is being used.

PRATT: We’ve spent the past decade in healthcare saying, “patients first,” yet the patient generally has the quietest voice. We must find ways to redesign the whole process with patients’ concerns in mind, protecting them and ensuring they’re educated about their rights.