Evaluation of a Vaccine Chatbot on HPV Vaccine Confidence and Hesitancy

A randomized controlled trial with 2 arms will be conducted to evaluate the effectiveness of an LLM-powered chatbot on improving HPV vaccination among females aged 15 to 26, followed by a pragmatic implementation science study to assess the public acceptance of chatbot and to identify the facilitators and barriers to its implementation in a real-world setting. The sample size is calculated based on the primary outcome of the HPV vaccination uptake among females aged 15 to 26 in China. This uptake is around 10% according to the China CDC estimation, and we assume that it can increase to 15% following the intervention. With a significance level (α) of 0.05 and 80% power (1-β=0.8), a sample size of at least 683 per group is needed to detect a 5% absolute difference in vaccination rate (from an assumed baseline rate of 10% to 15% post-intervention). The sample size is inflated to 900 per group to account for potential loss to follow-up and to ensure adequate power for a multi-center design. Participant recruitment will be conducted across three representative regions of China. For those aged 15 to 18, their parents will be also enrolled. A randomization method will be employed to ensure balanced allocation across two groups, and the randomization sequence will be generated by an independent statistician to ensure a 1:1 allocation ratio. Participants in the intervention group will be invited to use the HPV vaccine chatbot. The chatbot powered by a large language model (LLM), offers a scalable, accessible, and interactive platform around the HPV vaccine information. The control group will not get access to the vaccine chatbot until the end of the trial. Following the intervention, all participants will complete a questionnaire survey on their confidence, literacy, and hesitancy on the HPV vaccine. And their vaccination status will be obtained from official vaccination records at the end of trial. The intention-to-treat (ITT) analysis will be used to evaluate the effectiveness of the intervention. Between-group comparisons will use t-tests for continuous variables and chi-square tests for categorical variables. To adjust for potential confounders, generalized estimating equation (GEE) models will be employed to provide robust effect estimates. Timing of vaccination will be evaluated using Kaplan-Meier survival curves and log-rank tests, with Cox proportional hazards models to estimate intervention effects on time-to-vaccination. Subgroup analyses will assess heterogeneity of intervention effects across populations stratified by baseline characteristics.