A patient-facing conversational AI system called Articulate Medical Intelligence Explorer (AMIE) was evaluated during urgent primary care visits in a real-world setting. Findings from the prospective, single-center, single-arm feasibility study were published in The Lancet. 

“Our findings show that AMIE is conversationally safe within this physician-supervised, single-complaint, urgent-care setting study design. The safe presentation of potential diagnoses to patients suggests that conversational AI can meaningfully shift patient–AI interactions from simple information gathering to collaboration and counselling,” wrote the study authors, including corresponding author Adam Rodman, MD, of the Department of Medicine, Beth Israel Deaconess Medical Center.  

Background

Previous research from Google DeepMind and Google Research has demonstrated the potential of patient-facing AI to support patient intake and care coordination processes, potentially reducing the burden on health-care workers.  

Researchers previously evaluated AMIE’s capabilities in text-based diagnostic conversations and multivisit case scenarios to assess its potential role in clinical workflows.  

The current study was considered the first preregistered study of a patient-facing conversational diagnostic tool in a real-world care setting and was among the first to evaluate a system that presented potential diagnoses directly to patients.  

Study and Model Methods

The researchers conducted a prospective, single-center, single-arm feasibility study in which AMIE was used in a real-world clinical setting before urgent primary care visits. The study was conducted at Healthcare Associates, part of Beth Israel Deaconess Medical Center in Boston. All patients were adults, had a single chief complaint, and had been determined not to require emergency care. Patients with known pregnancy or mental health complaints were excluded. Each patient was offered up to $50 to participate in the study with AMIE and complete a subsequent user experience interview.  

Participating primary care clinicians included attending physicians, nurse practitioners, and internal medicine residents. For the latter, final management plans were developed under the supervision of an attending physician.  

Remote patient interactions with AMIE were scheduled 0 to 5 days before the planned urgent care visit. Before interacting with AMIE, patients completed a survey collecting demographic information, health literacy, technology literacy, and other information to assess their overall attitudes toward AI use in health care. During the session, AMIE conducted a clinical conversation with the patient without access to their medical chart. At the end of the session, AMIE presented information about potential diagnoses.

 The interactions were monitored in real time by a board-certified internal medicine physician, designated as the AI supervisor, to ensure patient safety. The supervisor could intervene and interrupt the conversation if necessary. Immediately after the session, the supervisor debriefed the patient to clarify information and correct any AI hallucinations. The patient and supervisor then each completed a survey.  

Transcripts of the interactions and AI-generated summaries were provided to the primary care clinicians before the urgent care visits. After the visits, patients and their primary care clinicians each completed another survey assessing visit efficiency and attitudes toward AI. All participants were also asked to complete an interview about their experiences.  

Eight weeks after the urgent care visits, a panel of three internists reviewed the medical records and determined a reference diagnosis. Two of the evaluators used a standardized scale for assessing differential diagnosis generators to evaluate the diagnostic accuracy of AMIE. These evaluators also conducted blinded, randomized reviews of patient management plans generated by AMIE and the primary care clinicians.  

The primary objective of the study was to evaluate the safety and quality of AMIE’s clinical conversations, with a focus on the number and types of safety stops. Secondary objectives included assessing patient and clinician perceptions of patient–AI interactions and the quality of AMIE’s diagnostic and management reasoning.

AMIE was built on Gemini 2.5 Pro without any domain-specific fine-tuning. The underlying model was subsequently switched to Gemini 2.5 Flash to reduce latency because of computational limitations in the study’s research infrastructure. The study protocol permitted such model changes.  

Results

Among 1,452 scheduled urgent care visits at Healthcare Associates during the study period, 114 patients initiated an AMIE encounter, and 100 completed the interaction (88%). Two patients were subsequently excluded because they did not attend a follow-up primary care visit.  

The median interval between the AMIE interaction and the primary care visit was 1 day.  

“We found the deployment of AMIE in a real patient care workflow to be practical with a high patient–AMIE interaction completion rate and follow-up to scheduled urgent care appointments,” the study investigators noted.  

Primary care clinicians had lower survey completion rates than patients (61% vs 90%–92%), with 27% of clinician surveys indicating that the clinician had not had an opportunity to review the AMIE interaction before the patient appointment.  

The included patients were generally representative of the clinic’s broader urgent care population during the study period, although participants tended to be younger than the overall clinic population.  

No conversation stops were required by the AI supervisor during the AMIE interactions. One hallucination was identified in which AMIE incorrectly reported a past surgery date as occurring in the future.  

Supervisors provided additional clinical information in five cases, including reassurance regarding malignancy as a potential diagnosis in three cases, further clarification of symptoms in one case, and clarification of criteria for seeking emergency care in the remaining case.  

The management plans generated by AMIE and primary care clinicians were rated as equal in quality in 48% of cases. Evaluators preferred AMIE’s plans in 6% of cases and the primary care clinicians’ plans in 27%.  

The safety of AMIE’s management plans was rated as favorable or very favorable in 93% of cases, compared with 95% for the primary care clinicians’ plans. For appropriateness, the corresponding rates were 83% and 94%, respectively. Cost-effectiveness was rated as favorable or very favorable for 85% of AMIE’s plans vs 97% of the clinicians’ plans. All primary care clinicians’ plans were rated as practical or very practical, compared with 94% of AMIE’s plans.  

The AI system included the adjudicated reference diagnosis among its top seven diagnostic candidates in 90% of cases and among its top three in 75% of cases. Its top-ranked diagnosis was correct in 56% of cases. AMIE’s diagnostic accuracy was higher in cases in which the adjudicated reference diagnosis was determined without a diagnostic test (n = 52).  

Regardless of whether patients initially expressed positive or negative attitudes toward AI, their attitudes generally improved after interacting with AMIE. Regarding the quality of the conversations, most patients found the AI system to be polite, attentive, and able to put them at ease. However, fewer patients expressed confidence that their information would be kept confidential or considered the AI system to be honest and trustworthy.  

Primary care clinicians rated the AI system favorably across most aspects of the encounters, including gathering information about patients’ chief complaints and medical and medication histories. However, AMIE received lower ratings for eliciting family history. Clinicians also indicated that AMIE performed well in explaining relevant clinical information and addressing patient concerns across all evaluated criteria.  

Among primary care clinicians who completed the surveys and reviewed the AMIE encounter before the urgent care visit, 75% found the interaction helpful in preparing for the appointment (n = 44 patients). The AI system was considered somewhat harmful in only one case, in which lymphoma was included among the potential diagnoses, causing distress that the patient subsequently reported during the primary care visit. Clinicians indicated that AMIE may have influenced their clinical behavior in 25 of 44 cases. They also expressed concern about premature diagnostic closure related to the AMIE encounter in three cases.  

Limitations

The study authors acknowledged the pilot nature and single-arm design of the study as key limitations. They cautioned that ratings may have been inflated by the novelty of the technology, financial incentives, and the lack of a comparator group. Additionally, the generalizability of the findings may be limited because participants tended to be younger than the overall urgent care population and certain patient groups were excluded.  

The authors also noted that many patients are not accustomed to accessing health care through a computer, and mobile device interactions with AMIE were not offered during the study.  

Furthermore, the presence of an AI supervisor during the interactions may have discouraged participants from testing the system’s boundaries or presenting external evidence related to their symptoms.  

The study authors noted that the findings provide initial real-world evidence of AMIE’s clinical reasoning performance. “These results represent a critical feasibility demonstration, which provides a foundation for future larger-scale research towards the integration of advanced conversational AI into clinical, patient-facing workflows to enhance care delivery,” they concluded.  

Insights

In an accompanying comment article, Mahmud Omar, MD, and Girish N. Nadkarni, MD, MPH, CPH, both of the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai, raised additional concerns about the study’s limitations. 

Drs. Omar and Nadkarni highlighted the potential for automation bias, noting that patients could become fixated on an AI-suggested diagnosis, even after the encounter, while clinicians could be influenced by the model’s framing of the case, potentially resulting in patient harm.  

They also emphasized that the level of human oversight incorporated into the study would be difficult to sustain on a larger scale. Instead, they recommended implementing automated safeguards.  

“Brodeur and colleagues should be commended on moving the field beyond simulated performance towards the harder question of what happens after an AI system names a diagnosis. Their study suggests that safety cannot be inferred from diagnostic accuracy, favourable ratings, or absence of real time intervention alone. It must include delayed patient effects, clinician anchoring, equitable access, model version changes, and the consequences of different oversight designs,” Drs. Omar and Nadkarni concluded. “The next generation of studies should therefore treat communication and allocation of attention as clinical interventions, not interface features.” 

DISCLOSURES: The study was funded by Alphabet. For full disclosures of the study authors, as well as access to limited open-source data sets, visit thelancet.com.  

Dr. Nadkarni reported consulting fees from Renalytix; is a scientific co-founder of Renalytix (minimal equity); and holds patents licensed to Renalytix, a company that develops and commercializes a U.S. Food and Drug Administration–authorized prognostic test for diabetic kidney disease using biomarkers and clinical variables, and that does not develop conversational or diagnostic dialogue AI. He is also a scientific co-founder of, and holds equity in, Pensieve Health, a genomics company focused on drug discovery that does not develop conversational clinical AI. Dr. Omar reported no competing interests.  

ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO^®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.