News Research AI in Oncology

Agentic LLM System May Improve Detection and Assessment of Immunotherapy-Related Adverse Events

November 17, 2026 Julia Cipriano 5 min read
Share Share via Email Share on Facebook Share on LinkedIn Share on Twitter

Researchers developed and evaluated an agentic large language model (LLM) system designed to identify and characterize immune-related adverse events documented in clinical notes of patients receiving immune checkpoint inhibitors. In a multiphase validation study, the findings from which were published in Nature Cancer, the system demonstrated the ability to detect them in retrospective testing. Additionally, AI-assisted review reduced annotation time and improved accuracy and agreement among clinical research staff.

Danielle S. Bitterman, MD
Danielle S. Bitterman, MD

Immune-related adverse events occur in up to 40% of patients receiving immune checkpoint inhibitors, but evidence of these toxicities is often buried within clinical notes, explained ASCO AI in Oncology editorial advisory board member Danielle S. Bitterman, MD, Vice President of AI for Clinical Development and AI for Science Innovation at AstraZeneca and Assistant Professor at Harvard Medical School in Boston, and colleagues.

Against this backdrop, they wrote, “The ability to surface candidate events with attributed causality and provisional grade shortly after documentation can shorten the time to recognition for life-threatening [immune-related adverse events] and standardize ascertainment for research registries and safety reporting.”

"We demonstrated the potential of agentic AI to improve both the quality and efficiency of [immune-related adverse event] curation, which could enhance our understanding and optimization of the benefits and risks of cancer treatment via clinical trials and real-world evidence generation," the study authors added. 

Study and Model Methods

The agentic LLM pipeline extracted information on six immune-related adverse event types—myocarditis, dermatitis, thyroiditis, hepatitis, colitis, and pneumonitis—from unstructured electronic health record notes. The system was designed to identify the presence of these events, distinguish current from past events, assign severity grades according to the Common Terminology Criteria for Adverse Events (CTCAE) v5.0 criteria, assess clinician-documented attribution to immune checkpoint inhibitors, and determine the certainty of that attribution.

The default agentic architecture routed clinical notes through an Event Splitter module to condition-specific extraction pipelines, each consisting of four sequential stages. For detection and grading, the system used a self-consistency approach involving three independent model runs and a judge agent to reconcile their outputs. Attribution and certainty were evaluated through single-agent assessments. The system also extracted text snippets directly from clinical notes to provide supporting evidence for its determinations at each stage of the workflow and assist human reviewers with annotation.

To identify the optimal configuration for note-level immune-related adverse event extraction, the investigators benchmarked proprietary and open-source LLMs across agentic and nonagentic architectures. Of the evaluated model–architecture combinations, GPT-4.1-mini with the default agentic architecture was found to demonstrate the strongest overall performance and was used for subsequent validation.

The study consisted of three phases conducted at a single health system. Retrospective development and evaluation involved 289 clinical notes, including 263 in the test set. The system was subsequently integrated into the electronic health record workflow for a 3-month prospective silent validation involving 884 clinical notes. Finally, a randomized crossover study involving 17 clinical research staff members reviewing 20 unique clinical notes compared AI-assisted annotation with standard manual review.

Key Findings

In retrospective testing, the GPT-4.1-mini–based agentic system achieved a macroaveraged F1 score of 0.92 for detection of current immune-related adverse events and 0.66 for multiclass severity grading. Performance varied by toxicity type, with dermatitis and thyroiditis achieving the highest performance and colitis achieving the lowest. Removing self-consistency reduced detection performance by 0.14 F1 points. The best-performing configuration, which retained this mechanism, had an estimated inference cost of approximately $0.02 per clinical note.

During prospective silent deployment, the investigators reported binary F1 scores across grading, attribution, and certainty labels of 0.72 for current events and 0.79 for past events, “underscoring the gap between retrospective evaluation and changing real-world conditions.”

In the randomized crossover study, AI assistance reduced median annotation time by 43.5%, from 428 seconds with manual review to 242 seconds with AI-assisted review. The model-estimated reduction was 2.71 minutes (P < .001).

AI assistance also increased complete-match accuracy, defined as requiring all current immune-related adverse event grades within a note to exactly match the expert-derived gold standard, from 19.4% to 25.9% (odds ratio = 1.45; P = .045). Interannotator agreement for current grade labels also improved, with Krippendorff's alpha increasing from 0.22 to 0.85 in one note set and from 0.51 to 0.82 in the other.

“An agentic LLM system and associated interactive interface, validated across retrospective and prospective settings, can extract real-time, clinically detailed information about the presence, severity, and attribution of [immune-related adverse events],” the investigators concluded. “Agentic assistance improved the efficiency, performance, and consistency of [immune-related adverse event] curation when used with a human-in-the-loop.”

Looking ahead, they stated, “With external validation and outcome-focused trials, such systems could enhance and accelerate real-time safety monitoring, trial curation, and high-reliability oncology care.”

DISCLOSURES: The authors acknowledged financial support from the National Cancer Institute, the American Cancer Society and American Society for Radiation Oncology, a Patient-Centered Outcomes Research Institute (PCORI) Project Program Award, and the Woods Foundation. Dr. Bitterman is Associate Editor for JCO Clinical Cancer Informatics, Annals of Oncology, and Radiation Oncology of HemOnc.org; serves on the scientific advisory board of Mercurial AI and Blue Clay Health; has received consulting fees from Inspire Exercise Medicine; and is employed by AstraZeneca. For full disclosures of the other study authors as well as data and code availability, visit nature.com.

ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.

KOL Commentary
Watch

Related Content