Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery

Purpose ChatGPT is a large language model trained on a large dataset covering a broad range of topics, including the medical literature. We aim to examine its accuracy and reproducibility in answering patient questions regarding bariatric surgery.Materials and methods Questions were gathered from nationally regarded professional societies and health institutions as well as Facebook support groups. Board-certified bariatric surgeons graded the accuracy and reproducibility of responses. The grading scale included the following: (1) comprehensive, (2) correct but inadequate, (3) some correct and some incorrect, and (4) completely incorrect. Reproducibility was determined by asking the model each question twice and examining difference in grading category between the two responses.Results In total, 151 questions related to bariatric surgery were included. The model provided comprehensive responses to 131/151 (86.8%) of questions. When examined by category, the model provided comprehensive responses to 93.8% of questions related to efficacy, eligibility and procedure options; 93.3% related to preoperative preparation; 85.3% related to recovery, risks, and complications; 88.2% related to lifestyle changes; and 66.7% related to other. The model provided reproducible answers to 137 (90.7%) of questions.Conclusion The large language model ChatGPT often provided accurate and reproducible responses to common questions related to bariatric surgery. ChatGPT may serve as a helpful adjunct information resource for patients regarding bariatric surgery in addition to standard of care provided by licensed healthcare professionals. We encourage future studies to examine how to leverage this disruptive technology to improve patient outcomes and quality of life.

Automated summary

This study evaluated the accuracy and reproducibility of the large language model ChatGPT in answering patient questions about bariatric surgery. By collecting questions and having them graded by certified bariatric surgeons, it was found that ChatGPT provided accurate and reproducible responses. Therefore, ChatGPT can serve as a helpful adjunct information resource for patients regarding bariatric surgery.