Two papers published simultaneously in Nature move agentic AI from the sidelines of clinical care toward a more active role within the clinical workflow. Both systems were evaluated in controlled, simulated environments and demonstrated clinical reasoning on par with that of comparator physicians. The findings have generated both enthusiasm and caution among clinicians and medical AI researchers.  

One paper, from corresponding author Jakob Nikolas Kather, MD, MSc, and colleagues, described a medical AI agent called MIRA, which was evaluated in a controlled, sandboxed virtual electronic health record (EHR) environment to assess its performance in clinical decision-making and structured operations within an emergency department workflow. The second, from researchers at Google DeepMind and Google Research, described expanded capabilities for AMIE, a conversational agentic system for disease diagnosis.  

Together, these companion papers mark a shift in what medical AI systems are being asked to do, moving beyond answering isolated clinical questions to assuming responsibility across an entire episode of care, from history-taking through treatment decisions and follow-up. Both research teams emphasized, however, that prospective validation in real-world settings will be required before either system is used in patient care. 

Medical Intelligence for Reasoning and Action (MIRA) 

Researchers from Germany sought to overcome prior challenges in developing autonomous large language models for health-care settings by evaluating their system within a controlled, sandboxed virtual EHR environment. 

Study Methods 

MIRA, which was designed to comply with Fast Healthcare Interoperability Resources (FHIR) standards and six additional coding systems, was evaluated on more than 500 cases from the publicly available MIMIC-IV data set within a simulated emergency department workflow. The agent made diagnostic and therapeutic decisions for patients in surgery, internal medicine, and oncology.  

A patient agent, designed to generate responses based on the patient’s history of present illness, interacted with MIRA. The medical agent then ordered and interpreted diagnostic tests and procedures. MIRA’s performance was compared with that of four board-certified physicians and, separately, with that of a mixed-seniority cohort comprising four residents and two board-certified physicians, reflecting typical staffing in German emergency departments.  

The agent’s performance was evaluated across eight disease areas: cholecystitis, pulmonary embolism, diverticulitis, appendicitis, pancreatitis, pneumonia, pancreatic cancer, and urinary tract infection. 

Model Methods 

MIRA was developed using OpenAI’s GPT-4o with a large language model for interpreting patient responses and test results and executable software. The agent was designed to perform 12 tasks that simulate clinical workflows, including dialogue and history-taking with the patient, ordering imaging studies and laboratory tests, making diagnoses, and determining patient disposition. Throughout the process, the medical AI agent interacted with the patient agent via chat.  

The medical agent also generated FHIR requests that were sent to the sandboxed EHR system. FHIR observations, or test results, were then returned to MIRA for interpretation and subsequent tasks, including medication requests.  

The patient agent was prompted to provide consistent responses based on the patient’s history of present illness without revealing diagnostic conclusions prematurely.  

Results 

The patient agent provided content-consistent responses in 99.4% of cases based on human assessment and in 98.9% of cases based on assessment by a large language model judge. All 933 cases were audited for diagnostic information disclosure (0%) and prior workup disclosure (3.3%). Prior workup disclosure occurred most frequently in pancreatic cancer cases (22.2%). When the patient agent was tested for resistance to hijacking attempts, prior workup disclosure increased to 13.5%, again occurring most frequently in pancreatic cancer cases.  

MIRA achieved an average diagnostic accuracy of 88.9% across the eight disease categories. The medical AI agent performed best in cases of appendicitis (98.6%) and pancreatitis (92.3%). Accuracy was lowest for pneumonia (72.4%) and urinary tract infection (77.6%).  

“[T]his comparison has limitations: some elements of the clinical context at presentation can remain undocumented, so comparing against a single dataset ‘ground-truth’ label can misrepresent the diagnostic performance,” the study authors, including corresponding author Jakob Nikolas Kather, MD, MSc, of the Department of Medical Oncology, National Center for Tumor Diseases, Heidelberg University Hospital, noted.  

MIRA achieved an average diagnostic accuracy of 87.8%, compared with 78.1% for the board-certified physicians (P < .001) and 71.1% for the mixed-proficiency cohort (P < .001). The largest difference in accuracy was observed for pancreatitis, with MIRA achieving 95.2% accuracy compared with 78.6% among the board-certified physicians and 61.9% in the mixed-proficiency cohort.  

MIRA followed a physician-like, stepwise workflow from patient presentation through admission, and its decisions closely mirrored routine clinical practice at each stage. However, MIRA was more likely than physicians to request physical examinations, except in pancreatic cancer cases, in which admission was often planned. The agent also ordered more blood tests across all diagnostic categories, whereas physicians tended to request slightly more imaging studies.  

“However, even with this higher coverage relative to physicians in our interface, MIRA still requested only about half of the laboratory analytes obtained in routine care in MIMIC-IV, indicating utilization below the dataset baseline rather than an ‘order-everything’ strategy,” the study authors noted.  

The researchers also assessed procedure match rate to determine whether the agent appropriately recommended medical procedures. MIRA consistently demonstrated greater recall, recommending 53.5% of relevant procedures across the eight disease categories vs 38.3% for the board-certified physicians. Overall, MIRA showed greater agreement with the data set baseline than the physician comparators.  

The medical AI agent’s prescribing decisions were also evaluated for alignment with clinical practice guidelines. MIRA demonstrated greater adherence to best practices than either physician comparator cohort. The agent was more likely than physicians to prescribe intravenous fluids for pancreatitis, although it did not fully adhere to guideline recommendations for antibiotic prescriptions.  

Next, the researchers evaluated patient-level prescribing and management risks across six safety domains: high-severity drug–drug interactions, renal dosing compatibility based on estimated glomerular filtration rate and creatinine, allergy–medication mismatches, QT-risk prescribing, unsafe opioid prescribing, and therapeutic duplication. The study authors identified no instances of high-severity drug–drug interactions, renal dosing incompatibilities, allergy–medication mismatches, QT-risk prescribing, or unsafe opioid prescribing. Therapeutic duplication was identified in three cases; however, the authors determined these instances to be clinically reasonable, although more explicit dosing instructions were needed.  

In terms of prescribing accuracy, MIRA generated 99.8% of tested prescriptions (n = 468) with relevant, clinically useful dosing instructions. Dosing was correct in 97.6% of cases, units were correct in 98.3%, treatment duration was correct in 99.6%, and the treatment period unit was correct in 99.8%. Administration frequency was considered appropriate in 99.1% of cases, and the route of administration was correct in 97%.  

The medical agent was also evaluated using predefined bias scenarios. Among more anxious patients, the risk difference was –1.2 percentage points, and when patient sex differed, the risk difference was –10.0 percentage points. Overall, MIRA maintained largely stable diagnostic performance across all bias scenarios.  

Limitations 

The study authors noted several limitations. Because patient interactions were simulated using history of present illness text extracted from discharge summaries, the conversations may have been more structured and medically precise than those occurring during typical emergency department encounters. In addition, the publicly available MIMIC-IV data set may have been included in the training data for the underlying large language model.  

The physician comparison also warrants consideration. Because there is no emergency medicine specialty in Germany, emergency departments are typically staffed by junior physicians under the supervision of board-certified physicians. Although MIRA was compared with two physician cohorts, the experience level of the junior physicians may have influenced the human benchmark scores.  

The authors emphasized that medical AI agents should not be designed to replace health-care professionals. Rather, they suggested these systems may be most effective in collaborative clinical environments, augmenting clinician decision-making under continuous human supervision. They concluded that autonomous AI agents such as MIRA will require rigorous prospective validation in real-world clinical settings before being considered for routine clinical use. 

Articulate Medical Intelligence Explorer (AMIE) 

In a companion paper, researchers from Google DeepMind and Google Research described updated diagnostic capabilities for AMIE, a large language model–based agentic system designed to support clinical conversations and disease management across multiple patient visits.  

Background 

AMIE was previously introduced as a conversational diagnostic AI system that demonstrated physician-level performance in text-based diagnostic conversations. The system has also been evaluated in an ambulatory primary care setting and for multimodal clinical reasoning.  

Updates to the model included replacing the PaLM 2 base model with Gemini 1.5 Flash to enable longer-context processing, and incorporating in-context retrieval with structured, guideline-grounded reasoning.  

Study Methods 

The researchers, including co-supervising and corresponding author Mike Schaekermann, PhD, of Google Research, conducted a randomized, blinded virtual Objective Structured Clinical Examination (OSCE) study. The trial compared the management reasoning of AMIE with that of 21 primary care physicians from Canada and India across 100 multivisit scenarios, each comprising three visits and spanning five clinical specialties. Cases were designed with prespecified ground-truth diagnoses based on guidance from the UK National Institute for Health and Care Excellence (NICE)  and BMJ Best Practice. Specialist physicians and patient actors evaluated and compared the performance of AMIE and the primary care physicians across multiple domains of management reasoning.  

During the study, physicians and AMIE completed the three virtual OSCE visits in randomized, blinded order. Patient actors used scenario packets for each case to simulate patients and communicated through synchronous text chat. Both the physicians and AMIE had access to the same clinical guidelines during and between visits. After each visit, both completed postvisit questionnaires that were evaluated by the patient actors and specialist physicians.  

To generate a management plan, the AI system analyzed the patient’s presentation, established management goals, and produced an annotated plan that included lab orders, diagnostic tests, treatments, and other recommendations, with references to source documents supporting its reasoning. The quality of the management plans was assessed across 15 evaluation domains, including investigations, treatment recommendations, error avoidance, precision, and adherence to clinical guidelines.  

The researchers also developed RxQA to benchmark the system’s medication reasoning. RxQA is a multiple-choice assessment derived from U.S. and UK national drug formularies and validated by board-certified pharmacists. In addition, they developed the Management Reasoning Empirical Key Features (MXEKF) rubric to evaluate management reasoning. Based on psychological research involving outpatient clinicians, the rubric assessed performance across 10 domains, including management of competing priorities, communication and shared decision-making, relationship building, and prognostication.  

Model Methods 

AMIE was designed as a large language model–based multiagent system comprising a patient-facing dialogue agent for multivisit conversations and a management reasoning agent that retrieves clinical guidance and generates care plans. The dialogue agent also orchestrates the activities of the other system components. Both agents were built on Gemini language models.  

Following supervised fine-tuning on simulated multivisit physician–patient conversations, medical question-answering tasks, EHR summarization tasks, and real-world medical dialogues, the model underwent reinforcement learning with human and AI feedback to improve its conversational and disease-management capabilities.  

To respond to the patient actor, the dialogue agent used a chain of reasoning that maintained continuity across multiple visits. It first generated a response plan, including follow-up questions or answers for the patient actor, and then drafted and refined its response to meet predefined quality standards.  

The updated AMIE system also incorporated an agent state—a modular data structure that continuously updated the patient summary, differential diagnosis, and management plan through an asynchronous background reasoning process. 

The clinical-guideline corpus used in the study contained 627 documents comprising approximately 10.5 million tokens—more information than could fit within Gemini’s context window at one time. The management reasoning agent therefore generated up to five natural-language searches and used a retrieval system to identify the most relevant documents. It was allocated 256,000 tokens for external clinical knowledge, allowing it to consider an average of approximately six guidelines simultaneously. The agent then generated four draft management plans before refining and merging them into a final plan. 

Results 

AMIE’s management reasoning capabilities were noninferior overall to those of the primary care physicians in the OSCE study. Management plans generated by AMIE performed as well as, or better than, those generated by physicians across all 15 evaluation domains and all three visits. Although the primary care physicians scored numerically higher than AMIE on several domains, none of the differences reached statistical significance.  

The agentic system scored significantly higher than the physicians for overall management plan appropriateness (visit 1: 95% vs 72%; P < .001; visit 2: 96% vs 80%; P = .002; visit 3: 98% vs 81%; P < .001) and treatment recommendations (visit 1: 87% vs 66%; visit 2: 90% vs 62%; visit 3: 94% vs 71%; P < .001 for all visits). During one or both follow-up visits, AMIE also scored significantly higher than physicians for appropriate recommendations, freedom from significant errors, appropriate follow-up recommendations, and avoidance of inappropriate treatments.  

The only domain in which AMIE scored below the primary care physicians was avoidance of inappropriate investigations during the third visit (88% vs 90%; P = .841).  

AMIE also outperformed the physicians in treatment precision (P < .001 for all three visits) and investigation precision (P < .001 for visits 2 and 3). Although both AMIE and the physicians scored highly for selection of applicable clinical guidelines, AMIE scored significantly higher for recommending treatments aligned with those guidelines (P < .001 for visits 2 and 3). The agentic system achieved a score of 100% for providing explicit references to clinical guidelines across all visits.  

For the MXEKF evaluation, both specialist physicians and patient actors preferred AMIE across several assessment domains. The median tie rate between AMIE and the primary care physicians was 49% (95% confidence interval [CI] = 37%–62%). When evaluators expressed a preference, AMIE was favored a median of 47% of the time (95% CI = 33%–58%) vs 7% for the physicians (95% CI = 2%–13%). 

From the patient actor’s perspective, preferences for AMIE became more pronounced over successive visits in the domains of “monitoring and adjustment of the management plan” and “organization of the clinical encounter.”  Among specialist physicians, preference for AMIE’s relationship-building abilities also increased over time, whereas physician performance remained consistent.  

In terms of medication reasoning, AMIE outperformed physicians on the more difficult questions (n = 318 vs 282 lower-difficulty questions) under both closed- and open-book conditions, although both AMIE and the physicians improved when given access to external information. No significant differences were observed for the lower-difficulty questions in either setting.  

“The use of AMIE in longitudinal care settings, including successful interpretation of further diagnostic information such as laboratory results and information from other medical consultants, was an important test of a potential real-world use case for conversational AI,” the study authors wrote. 

Limitations 

The study authors acknowledged several limitations. Patient actors cannot truly replicate real-world clinical care, and the case mix was not representative of routine clinical practice. The study also differed from real-world practice by providing both AMIE and the primary care physicians with selected clinical guidelines that did not correspond to the physicians’ countries of practice. In addition, the interval between visits was substantially shortened, likely improving physicians’ recall of individual cases. Finally, the text-only interface limited communication by eliminating intonation, body language, and other audiovisual cues.  

The authors also suggested that some limitations of the system, including confabulation rates, may improve as the underlying large language model continues to advance, noting that better performance was observed when AMIE was tested with a newer backbone model.  

The authors concluded that prospective clinical studies will be required before the system can be considered for real-world clinical use. 

Takeaways 

The authors of the MIRA paper carefully positioned their system as a clinical decision-support tool rather than a replacement for clinicians. They anticipated that early real-world adoption would “focus on more narrowly bounded, high-volume tasks (such as reconciling pre-admission medications, assembling laboratory test panels, auto-drafting inter-department consultation requests, or suggesting guideline-conforming orders) under explicit physician review, freeing clinicians to redirect their attention to direct patient interactions,” Ferber et al wrote. 

The AMIE paper, meanwhile, raised the possibility that conversational AI could help address care fragmentation and continuity gaps. “Our results indicate that AMIE and other conversational management agents could eventually provide continuity in otherwise fragmented health systems, either independently or working with clinicians. If such systems could be developed and tested with care and robust clinical evidence, they would have the potential to address growing unmet clinical needs caused by global shortages and inequalities of supply of PCPs, which are exacerbated by concerns of physician burnout and increasingly complex populations under management,” Liévin et al wrote.  

“[W]e believe that the strategies behind AMIE and MIRA emphasize complementary strengths—evidence retrieval during dialogue and workflow automation inside the EHR—and are best viewed as synergistic approaches to clinically integrated AI,” Ferber et al noted. A mature clinical AI system would likely require both capabilities: the conversational continuity demonstrated by AMIE and the operational fluency demonstrated by MIRA. For oncology, where treatment decisions rely on complex, rapidly evolving guidelines and care coordination across specialties is routine, these complementary capabilities could prove particularly relevant. 

Even so, both research teams emphasized that neither system is ready for clinical use and that further evaluation will be required before either can be deployed in a real-world clinical setting.  

Responses and Insights 

The two papers prompted a widespread reaction, drawing both enthusiasm and caution from clinicians and AI experts.  

Eric Topol, MD, of the Scripps Research Translational Institute, called the studies “a major step forward within the constraints of a simulation, not real medicine.” He cautioned that the systems’ close alignment with clinical guidelines—a central finding of both papers—may not be an unqualified benefit. “Frankly, over-adherence to guidelines may presage the loss of the art of medicine, not taking in the human factor of each patient, and the human-to-human bond that would be the foundation of the patient-doctor relationship,” Dr. Topol wrote. Proving real-world value, he added, would ideally require randomized trials comparing end-to-end medical AI, human clinicians alone, and a combination of the two. 

Jorge Reis-Filho, MD, PhD, FRCPath, Chief of AI for Science Innovation at AstraZeneca, also acknowledged the progress represented by the studies in a LinkedIn post. “This is a remarkable step for agentic AI in clinical practice,” he wrote. Dr. Reis-Filho also questioned what it would take to move the research into prospective studies: “What will it take for hospitals to benchmark pathway-level medical intelligence prospectively, safely and at scale?” 

In expert commentary provided to the UK Science Media Centre, Julie Jacko, PhD, Chaired Professor of Health Informatics and Data Science at The University of Edinburgh, noted that the evaluation of the AMIE system was designed around guideline concordance, whereas physicians are not constrained in the same way, making the comparison “somewhat asymmetric.” While Dr. Jacko called the MIRA paper “ambitious and technically rigorous,” she also noted that the model was being “rewarded for reproducing recorded clinical behavior rather than necessarily demonstrating optimal care” and that ordering more tests or procedures is not the same as demonstrating better clinical judgment.  

Alfonso Valencia, PhD, ICREA Professor and Director of Life Sciences at the Barcelona Supercomputing Center, also noted that although MIRA is open source, AMIE is not, making it impossible for independent researchers to evaluate the system.  

Stefano Natangelo, MD, of the Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, also commented in a LinkedIn post that the two papers address the evaluation gap in clinical AI in different ways. “AMIE attaches grounding to the final answer. MIRA constrains the vocabulary of possible actions. Neither mechanism proves that the clinical decision itself was generated through a stable, reproducible process,” he wrote.  

“These are careful and ambitious papers. Probably among the strongest demonstrations so far of clinical AI agents moving toward real workflows. Precisely because they are strong, they expose the next problem more clearly,” Dr. Natangelo added. “Once the model becomes an agent, the unit of evaluation cannot just be the final answer. It has to include trajectory, repeatability, and the failure modes hidden when the output looks correct.” 

DISCLOSURES: MIRA: The work was funded by the European Union. Dr. Kather declared consulting services for Bioptimus, Panakeia, AstraZeneca, and MultiplexDx; shares in StratifAI, Synagen, and Ignition Lab; an institutional research grant from GSK; and honoraria from AstraZeneca, Bayer, Daiichi Sankyo, Eisai, Janssen, Merck, MSD, BMS, Roche, Pfizer, and Fresenius. For full disclosures of the other study authors, as well as data and code availability, visit nature.com.  

AMIE: This study was funded by Alphabet Inc. and/or a subsidiary thereof. Most of the study authors are employees of Alphabet and may own stock as part of their compensation package. For full disclosures of the study authors, as well as access to open-source data sets used as part of the research, visit nature.com. 

ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO^®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.