Zhengdong Wu

Importance. Persistent misinformation and low confidence impede human papillomavirus (HPV) vaccine uptake in Japan. Artificial intelligence (AI)-driven conversational agents may provide scalable vaccine education, but randomized evidence for HPV vaccine communication remains limited.
Objective. To determine whether an AI-driven HPV vaccine chatbot improves caregiver HPV vaccine literacy compared with a government leaflet.
Design, setting, and participants. Parallel-group randomized clinical trial (RCT) conducted through a national research panel of female caregivers in Japan with unvaccinated daughters aged 12 to 18 years from November 30 to December 27, 2024, with 2-week follow-up.
Interventions. Participants were randomized 1:1 to an AI-driven HPV vaccine chatbot or standard government HPV vaccine leaflet.
Main outcomes and measures. The primary outcome was an HPV vaccine literacy, evaluated immediately after intervention to estimate immediate intervention efficacy and evaluated at 2-week follow-up to estimate durability. HPV vaccine literacy was measured using 7 factual questions about HPV and HPV vaccination; scores are shown on a 0 to 7 scale, with higher scores indicating more correct responses.The secondary outcome was decision to vaccinate at 2-week follow-up.
Results. Among 848 randomized participants, 704 of 841 (83.7%) were included in the immediate modified intention-to-treat (mITT) analysis (353 chatbot, 351 leaflet) and 477 of 841 (56.7%) in the mITT analysis at 2 weeks (237 chatbot, 240 leaflet). A total of 447 of 703 participants (63.6%) were aged 40 to 49 years and had college or university education (472 of 703 participants [67.1%]). Immediately after the intervention, analysis of covariance-adjusted literacy was higher with the chatbot (adjusted mean difference, 0.30 points on the 0 to 7 scale; 95% CI, 0.18-0.43 points; P < .001). At the 2-week durability assessment, mean (SD) literacy increased by 0.40 (1.21) with the chatbot vs 0.28 (1.13) points with the leaflet; adjusted literacy again favored the chatbot (adjusted mean difference, 0.30 points; 95% CI, 0.13-0.47; P < .001). At 2 weeks, caregivers’ vaccination decisions did not differ between groups (102 of 253 participants [40.3%] vs 103 of 260 participants [39.6%]; adjusted odds ratio, 0.74; 95% CI, 0.41 to 1.33; P = .31).
Conclusions and relevance. In this RCT of female caregivers in Japan, an AI-driven HPV vaccine chatbot produced higher adjusted HPV vaccine literacy than a government leaflet immediately after intervention and at 2-week follow-up, without a measurable short-term effect on vaccination decision. Conversational AI may complement official materials and health care professional recommendations.
Trial registration. ClinicalTrials.gov Identifier: NCT06702423.

Large language models (LLMs) are increasingly used by clinicians and the public for vaccine information, yet their factual accuracy across languages and vaccine domains remains insufficiently characterized. We evaluated 13 LLMs using VaxEval, a multilingual vaccine-knowledge benchmark of 1886 vaccine-related multiple-choice questions spanning 14 vaccines in English (71%), Spanish (13%), and Chinese (16%). All items underwent quality control, with reference answers verified against authoritative guidance and peer-reviewed sources. Model performance was evaluated under zero-shot, few-shot, and chain-of-thought (CoT) prompting, with exact-match accuracy defined as selecting the pre-specified reference option. We used mixed-effect logistic regression to estimate associations between model group (newer flagship models vs earlier models), prompting strategy, language, and vaccine type, and answer correctness. Mean accuracy across models was 86.0% in English, 83.7% in Spanish, and 80.0% in Chinese. Flagship models had higher odds of correctness than earlier versions (OR 1.57; 95% CI 1.50-1.65; P < .001). Few-shot prompting was associated with higher correctness (OR 1.17; P < .001), whereas CoT prompting was associated with lower correctness (OR 0.79; P < .001). Performance varied by vaccine type and question category, underscoring the need for rigorous evaluation, structured guardrails, and targeted refinement before using LLMs for vaccine communication.

Conversational artificial intelligence, in the form of chatbots powered by large language models, offers a new approach to facilitating human-like interactions, yet its efficacy in enhancing vaccination uptake remains under-investigated. This study assesses the effectiveness of a vaccine chatbot in improving human papillomavirus (HPV) vaccination among female middle school students aged 12-15 years across diverse socioeconomic settings in China, where HPV vaccination is primarily paid out-of-pocket. A school-based cluster randomized trial was conducted from 18 January to 31 May 2024. The study included 2,671 parents from 180 middle school classes stratified by socioeconomic setting, school and grade level in Shanghai megacity, and urban and rural regions of Anhui Province. Participants were randomly assigned to either the intervention group (90 classes, 1,294 parents), which engaged with the chatbot for two weeks, or the control group (90 classes, 1,377 parents), which received usual care. The primary outcome was the receipt or scheduled appointment of the HPV vaccine for participants’ daughters. In intention-to-treat analyses, 7.1% of the intervention group met this outcome versus 1.8% of the control group (P < 0.001) over a two-week intervention period. In addition, there was a statistically significant increase in HPV vaccination-specific consultations with health professionals (49.1% versus 17.6%, P < 0.001), along with enhanced vaccine literacy (P < 0.001) and rumor discernment (P < 0.001) among participants using the chatbot. These findings indicate that the chatbot effectively increased vaccination and improved parental vaccine literacy, although further research is necessary to scale and sustain these gains. Clinical trial registration: NCT06227689 .

Chatbots have become an increasingly popular tool in the field of health services and communications. Despite chatbots’ significance amid the COVID-19 pandemic, few studies have performed a rigorous evaluation of the effectiveness of chatbots in improving vaccine confidence and acceptance. In Thailand, Hong Kong, and Singapore, from February 11th to June 30th, 2022, we conducted multisite randomised controlled trials (RCT) on 2,045 adult guardians of children and seniors who were unvaccinated or had delayed vaccinations. After a week of using COVID-19 vaccine chatbots, the differences in vaccine confidence and acceptance were compared between the intervention and control groups. Compared to non-users, fewer chatbot users reported decreased confidence in vaccine effectiveness in the Thailand child group [Intervention: 4.3 % vs. Control: 17%, P = 0.023]. However, more chatbot users reported decreased vaccine acceptance [26% vs. 12%, P = 0.028] in Hong Kong child group and decreased vaccine confidence in safety [29% vs. 10%, P = 0.041] in Singapore child group. There was no statistically significant change in vaccine confidence or acceptance in the Hong Kong senior group. Employing the RE-AIM framework, process evaluation indicated strong acceptance and implementation support for vaccine chatbots from stakeholders, with high levels of sustainability and scalability. This multisite, parallel RCT study on vaccine chatbots found mixed success in improving vaccine confidence and acceptance among unvaccinated Asian subpopulations. Further studies that link chatbot usage and real-world vaccine uptake are needed to augment evidence for employing vaccine chatbots to advance vaccine confidence and acceptance.