Although obtaining informed consent from patients so they understand the risks and benefits before undergoing a recommended therapy is standard practice in oncology care, the rapid integration of AI in clinical care is raising ethical and legal issues surrounding what oncologists are obligated to tell patients about AI use in their care. Informed consent in oncology is primarily governed by state laws and medical ethics, and generally requires that clinicians disclose information that is material to a reasonable patient’s decision to agree to the administration of a treatment plan. However, the “permeation of AI tools into health care is testing the traditional understandings of what patients should be told about their care,” according to Michelle Mello, JD, PhD, MPhil, Professor of Law and Professor of Health Policy at Stanford University.

Despite the importance of securing informed consent to ensure patients understand their diagnosis, treatment intent (curative or palliative), potential risks, and alternative options, clinical decision-support systems used in routine care are not subjected to traditional patient informed consent requirements. The question Dr. Mello posed in her JAMA Perspective “Ethical Obligations to Inform Patients About Use of AI Tools,” is whether AI-supported decision-making applications should be treated similarly.

The answer is complicated. Dr. Mello cites the results from several surveys suggesting that many patients have negative reactions to the use of AI in their care, including 60% who reported feeling uncomfortable with their physician relying on AI,up to 80% who had low expectations AI would improve important aspects of their care, and 63% who said it was very true that they would want to be notified about the use of AI in their care. 

In addition, a recent nationwide survey by the National Comprehensive Cancer Network (NCCN) Cancer Experience Registry of cancer survivors found that while 45% agreed that they can see the benefits of using AI in health care, 44% reported feeling scared of the technology in health-care settings.

In Dr. Mello’s JAMA Perspective, she and her colleagues offer some general guidance to assist oncologists in navigating this new technological terrain. They recommend that patients with cancer should be notified about the use of AI in their care and provide consent or receive notification when the tools entail potential physical risk and the patient has a meaningful opportunity to act based on the disclosed information. For example, consent would be appropriate for:

• Use of an AI-guided, nonautonomous surgical robot in their treatment because surgery typically involves considerable risk of harm, and patients can choose nonrobotic surgery instead; and

• Tools that analyze the genomic information of patients’ tumors to predict responses to specific drugs and that recommend treatment plans, because inaccuracy could lead to poor outcomes and patients can opt out of having this tool used in their care.

Dr. Mello also recommends oncologists notify patients when:

• Generative AI tools are used to draft emails to patients because inaccuracy may cause harm, and informed patients may be activated to question odd email replies; and

• Ambient listening tools are used to generate clinic visit summaries because inaccuracy may cause harm and patients can check summaries for accuracy.
  (Editor’s Note: Some states, including California and Texas, also have explicit legal mandates requiring clinicians to notify patients when certain types of AI tools are being used. Because ambient AI tools record audio conversations that are sent to third-party servers for processing, oncologists must also follow state wiretapping laws, and other laws relating to recording private conversations, which typically require two-party permission. The American Medical Association State Legislative Activity Brief offers additional information on state legislative activity concerning AI in health care.)

In this wide-ranging interview with ASCO AI in Oncology, Dr. Mello discussed the ethical obligations oncologists have to reveal how they are using AI tools in patient care, when oncologists should obtain patient consent when using ambient scribes, and how to reduce biases in AI models.

Assuring Patients AI-Powered Tools Are Useful and Safe

What are the ethical obligations for clinicians to reveal how they are using AI-driven models in patient care, especially in oncology, where AI is increasingly being used to diagnose and personalize treatment?

It’s an important question that everyone is trying to understand. The answer for me is that it depends on the use cases and the specific AI tool being used and how it is being used in clinical care. I spend a lot of time in conversations with patients here at Stanford to get their opinions about the AI tools we are thinking about using at the medical school. What they’ve told me, which was surprising, is that unless the information is useful for them to have, they don’t want it.

Patients say they are already dealing with a lot of information when they check into the hospital, and they just want assurances that clinicians are doing everything to keep them safe and well. They understand that providers are now using AI-powered decision-support tools in clinical care. What patients care most about in this technological environment is that at the end of the day, it’s the provider’s own medical judgement that is driving treatment decisions.

Trust in the physician and the hospital are the most important factors for patients. For many AI-powered tools that operate in the background of care, I’m not sure that a disclosure to the patient about how her oncologist is using these tools is ethically required. However, patients should be informed if their oncologist is supporting a treatment recommendation based in part on information gleaned from a clinical decision-support AI algorithm.

The notification creates a conversational opening for patients who want to know more about how the technology is being used in their care, and allows them to ask, “What’s your judgment, doctor?”

Separating the Ethical and Legal Obligation to Patients

The use of ambient AI scribes is becoming ubiquitous in oncology patient care. In the framework you have developed to help practitioners determine what their patients should be told about their clinical use of AI, you recommend that patients be informed that their clinician is using the tool. Why?

In many states, it is mandatory that clinicians tell their patients they are using ambient scribes. For example, in states that have wiretapping laws that require two-party consent for ambient recording during patient visits. It’s not an ethical matter; it’s a legal one.

In states that don’t have laws requiring notification, I still think it is useful for clinicians to inform their patients they are using this recording device, particularly in the outpatient context, because patients may want to check the notes from an office visit and make corrections. In the in-patient hospital setting, physicians may be both legally and ethically bound to get consent from patients if a recording device is turned on at all times, because patients are entitled to privacy.

So, to me, even if physicians aren’t legally obligated to get patient consent for ambient recording, it is ethically mandatory.

Reducing Bias in AI Algorithms

How can oncologists ensure the information from AI tools that they are giving marginalized and underserved patients is unbiased; what disclosures should oncologists give these patients about their care plan?

The most important conversations regarding the issue of bias in AI models has to happen between the hospital and the AI vendor to ensure that the training datasets used in these models are highly representative of the patient populations receiving cancer care. And, hopefully, the hospital has been conducting its own tests to evaluate the data for potential biases in their patient population, and to ensure that the AI models are aligned with institutional goals and standards.

It’s important to keep in mind that in addition to race, ethnicity, and sex/gender, there are other categories of patients that can cause underperformance of AI algorithms; for example, patients with frailty. Once hospitals adopt these tools, they should conduct continuous monitoring to evaluate how well these models are performing across different subpopulations of patients.

Finally, hospitals should educate clinicians about known weaknesses with the model. For example, scribe tools may not work as well for patients with accented English, so clinicians need to be especially vigilant when they evaluate AI output for certain groups of patients.

Choosing to Opt-Out of the Use of AI

What are the opportunities for patients to opt out of AI services if they choose; how can health-care organizations ensure they are adhering to their patients’ wishes for treatment?

We’ve discussed the legal and ethical obligations of informing and getting patient consent about the use of ambient scribes. It may also be appropriate to offer patients opt-out opportunities if alternative care modalities are available. For example, if patients prefer surgery that does not involve an AI-guided robot or mammography without AI interpretation.

In many other instances, it may not be realistic or practical to offer an á la carte menu of all the AI tools used in their care. For example, the hospital’s use of a predictive algorithm to determine whether to prestock blood in the operating room prior to surgery.

Given that opting out of these tools isn’t usually on the table, hospitals should honor patients’ interests by working hard to assure their safety and privacy protection. What is more important than just communicating the risks of AI tools is taking action to mitigate them.

DISCLOSURES: Dr. Mello receives grant funding from Stanford Health Care. She has received honoraria from AVIA Health and Augmedix, and her spouse is an executive at Cisco Systems, a manufacturer of platforms that power AI.

ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO^®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.