Researchers Propose Service Quality Index for AI Health Care Chatbots
A mixed methods study described the development of a service quality index system for AI health care chatbots, adapted from the SERVQUAL framework. The results published in the Journal of Medical Internet Research present a validated tool for evaluating chatbot performance and provide insights intended to enhance AI-driven medical consultation services.
“The proposed index system offers practical value for multiple stakeholders: it enables users to better understand and assess the strengths of AI health care chatbots; supports health service managers in systematically collecting feedback and monitoring performance; and guides developers in conducting feasibility analyses, optimizing design, and implementing postlaunch evaluation,” commented Yu Gu, PhD, of Capital Medical University, Beijing, and a study coauthor.
The finalized service quality index system comprises five primary and 17 secondary indicators. The primary indicators, listed in descending order of weight, are assurance (0.239), reliability (0.237), human-likeness (0.187), tangibility (0.170), and responsiveness (0.167). Distribution of the secondary indicators includes four (24%) for reliability, four (24%) for tangibility, three (18%) for assurance, three (18%) for human-likeness, and three (18%) for responsiveness.
The investigators wrote that at the study design stage, they expected their findings would “contribute to better identification of shortcomings, promote continuous quality improvement, enhance user experience, and offer new insights into the systemic evaluation of service quality of AI health care chatbots.”
How and Why the Index Was Developed
AI-powered chatbots are increasingly used in health care settings, where they may serve as AI physicians for online medical consultations. Yet despite widespread adoption, the investigators noted that there has been limited guidance on how to comprehensively evaluate their service quality.
To address this gap, they aimed to develop an index tailored specifically to AI health care chatbots. The system was grounded in SERVQUAL, one of the most widely recognized frameworks for evaluating medical service quality worldwide, encompassing five domains: tangibility, reliability, responsiveness, assurance, and empathy.
The investigators conducted a comprehensive literature review and consulted four domain experts to generate an initial pool of 26 second-level indicators for assessing the quality of AI health care chatbots. After a Delphi-based refinement and selection process, 17 indicators were retained. These indicators were organized into five domains adapted from the SERVQUAL framework, with human-likeness replacing the traditional empathy dimension to capture the ability to create personalized responses and attention to emotion.
The index system was finalized through a two-round Delphi process involving 20 experts recruited through purposive sampling from hospitals, universities, and health commissions. A virtual meeting was conducted between the two rounds. In a third round, the analytic hierarchy process was applied to assign weights to the indicators.
The response rate for both Delphi rounds and the analytic hierarchy process was 100%, and authoritative coefficients exceeded 0.7.
The investigators concluded, “This index system prioritizes user needs and experiences and can practically quantify the service quality of AI health care chatbots. The proposed index system will provide valuable support for health policymakers, service managers, and developers by enabling benchmark comparisons, facilitating quality monitoring, and guiding continuous service enhancement.”
Looking ahead, they stated, “Future research will involve applying this index system in field studies with users of AI health care chatbots to validate their utility and support its ongoing refinement.”
DISCLOSURE: The study was funded by the Social Science Fund of Capital Medical University. Dr. Gu and the other study author reported no conflicts of interest.
Share your own positive or negative experiences with AI health care chatbots in medical consultations services or tell ASCO AI in Oncology which quality measures you would like to see implemented in AI health care chatbots.
ASCO AI in Oncology is published by Conexiant under a license arrangement with the American Society of Clinical Oncology, Inc. (ASCO®). The ideas and opinions expressed in ASCO AI in Oncology do not necessarily reflect those of Conexiant or ASCO. For more information, see Policies.
Performance of a convolutional neural network in determining differentiation levels of cutaneous squamous cell carcinomas was on par with that of experienced dermatologists, according to the results of a recent study published in JAAD International.
“This type of cancer, which is a result of mutations of the most common cell type in the top layer of the skin, is strongly linked to accumulated [ultraviolet] radiation over time. It develops in sun-exposed areas, often on skin already showing signs of sun damage, with rough scaly patches, uneven pigmentation, and decreased elasticity,” stated lead researcher Sam Polesie, MD, PhD, Associate Professor of Dermatology and Venereology at the University of Gothenburg and Practicing Dermatologist at Sahlgrenska University Hospital, both in Gothenburg, Sweden.