Large language models (LLMs) demonstrated performance comparable to or better than that of trainees in assessing symptom severity during oncology telephone triage. Similar findings were observed for the generation of physician- and patient-facing drafts. Findings from the retrospective analysis were presented at the 2026 ASCO Quality Care Symposium (Abstract 672).  

“We found high sensitivity and negative predictive value of LLMs for severe cases, suggesting potential roles in screening and streamlining low-acuity cases,” said lead study author Luxiga Thanabalachandran, BSc, of the Cancer Centre of Southeastern Ontario, Queen’s University, Canada.  

“LLMs show promise as human-supervised tools within oncology telephone triage workflows to support more timely and accurate responses,” she added.  

Background and Study Methods

Nurse-led telephone triage of cancer symptoms is a key access point for supportive care between clinic visits. However, rising call volumes and increasing clinical complexity can make timely assessment and documentation challenging.  

LLMs could assist with acuity flagging and documentation drafting to support cancer care teams, but evidence evaluating these tools remains limited.  

The study aimed to assess whether LLMs could support oncology symptom telephone triage using real-world patient calls.  

Researchers conducted a retrospective, preimplementation analysis of 154 oncology telephone triage calls from the Cancer Centre of Southeastern Ontario between September 2024 and January 2026. They evaluated three LLMs—ChatGPT 5.5, Gemini 3.5, and Copilot—and three trainees—one postgraduate year 4 (PGY-4) fellow and two senior medical students. Two medical oncologists established the expert reference standard.  

Each LLM and trainee produced an acuity classification, a patient-facing message, and a physician-facing note.  

The key outcome measures were acuity concordance, direction of classification errors, draft quality (scored out of 20), and generation time. 

Study Findings

In terms of acuity performance, the three LLMs performed comparably to the three trainees. ChatGPT and Gemini each achieved 73.4% concordance with the expert reference standard, compared with 71.4% for Copilot. The fellow achieved 69.5% concordance, and the two medical students achieved 61% and 68.2%, respectively.  

“Importantly, when LLMs were discordant, they had a predominance towards over-triage compared to clinical trainees who were more likely to under-triage,” Ms. Thanabalachandran noted.  

Cases classified as severe by the experts were rarely missed by the LLMs. Copilot correctly identified 14 of 15 expert-rated severe cases, yielding a sensitivity of 93.3% and a negative predictive value of 99.1%. “The trade-off of this was false positive escalation,” Ms. Thanabalachandran said. Copilot classified 38 calls as severe, compared with 15 classified as severe by the experts, resulting in a positive predictive value of 36.8%.  

“This highlights a potential role for LLMs in identifying calls that are unlikely to be severe, to streamline the review and documentation of low-acuity cases for triage nurse verification,” she said.  

In terms of documentation quality, the LLMs outperformed the three trainees in generating both patient- and physician-facing notes. Copilot achieved the highest draft-quality scores, with mean composite scores of 18.0 for patient-facing notes and 18.4 for physician-facing notes. In comparison, the fellow achieved scores of 16.8 and 16.6, respectively.  

All three LLMs generated documentation substantially faster than the trainees, with mean response times ranging from 126 to 130 seconds, compared with 209 to 454 seconds for the trainees.  

Ms. Thanabalachandran explained that because the study was a retrospective, single-center analysis, external validation is needed to confirm the generalizability of the findings. In addition, the LLMs relied on voicemail transcripts rather than interactive triage, so they could not ask clarifying questions. The use of trainees as comparators also did not allow for direct benchmarking against the current nurse-led workflow. Furthermore, the tendency of LLMs to over-triage could also increase resource utilization by prompting additional urgent assessments and referrals and increasing staff workload.  

Ms. Thanabalachandran outlined a proposed workflow for the prospective use of LLMs in oncology symptom telephone triage: “Proposed prospective workflow includes identifying potentially low-acuity calls, drafting patient guidance and clinical notes for triage nurse review, and clinical action as appropriate.” 

“These findings support prospective evaluation of staff-supervised LLM assistants in oncology telephone triage workflows,” she concluded, emphasizing that these AI tools should not replace clinical judgment.  

DISCLOSURES: The study authors reported no conflicts of interest. 

ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO^®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.