The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries
Journal of Bone and Mineral Research, vol. 39, pp. 106–115
Abstract
Artificial intelligence (AI) chatbots utilizing large language models (LLMs) have recently garnered significant interest due to their ability to generate humanlike responses to user inquiries in an interactive dialog format. While these models are being increasingly utilized to obtain medical information by patients, scientific and medical providers, and trainees to address biomedical questions, their performance may vary from field to field. The opportunities and risks these chatbots pose to the widespread understanding of skeletal health and science are unknown. Here we assess the performance of 3 high-profile LLM chatbots, Chat Generative Pre-Trained Transformer (ChatGPT) 4.0, BingAI, and Bard, to address 30 questions in 3 categories: basic and translational skeletal biology, clinical practitioner management of skeletal disorders, and patient queries to assess the accuracy and quality of the responses. Thirty questions in each of these categories were posed, and responses were independently graded for their degree of accuracy by four reviewers. While each of the chatbots was often able to provide relevant information about skeletal disorders, the quality and relevance of these responses varied widely, and ChatGPT 4.0 had the highest overall median score in each of the categories. Each of these chatbots displayed distinct limitations that included inconsistent, incomplete, or irrelevant responses, inappropriate utilization of lay sources in a professional context, a failure to take patient demographics or clinical context into account when providing recommendations, and an inability to consistently identify areas of uncertainty in the relevant literature. Careful consideration of both the opportunities and risks of current AI chatbots is needed to formulate guidelines for best practices for their use as source of information about skeletal health and biology.
Authors 10
-
Affiliation as printed
Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States
-
Affiliation as printed
Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States
-
Affiliation as printed
Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States
-
The University of Sydney · Garvan Institute of Medical Research · UNSW Sydney
Affiliation as printed
School of Medicine Science , Faculty of Medicine and Health, , Sydney, 2006 , Australia
Skeletal Diseases Program, The Garvan Institute of Medical Research , Darlinghurst, 2010 , Australia
St Vincent’s Clinical Campus School of Clinical Medicine, University of New South Wales , Kensington, 2052 , Australia
The University of Sydney , Faculty of Medicine and Health, , Sydney, 2006 , Australia
School of Medicine Science, Faculty of Medicine and Health, The University of Sydney, Sydney, 2006, Australia
-
University of Auckland · UConn Health
Affiliation as printed
Center for Regenerative Medicine and Skeletal Development, School of Dental Medicine, UConn Health , Farmington, CT 06030 , United States
Department of Molecular Medicine and Pathology, University of Auckland , Auckland, 1142 , New Zealand
-
Annegreet G. Veldhuis‐Vlug Aachen Center for Bone Quality Department of Internal Medicine Center for Bone Quality Department of Internal Medicine
Leiden University · Leiden University Medical Center
Affiliation as printed
Center for Bone Quality , Department of Internal Medicine, , Leiden, 2300 , The Netherlands
Leiden University Medical Center , Department of Internal Medicine, , Leiden, 2300 , The Netherlands
Center for Bone Quality, Department of Internal Medicine, Leiden University Medical Center, Leiden, 2300, The Netherlands
-
Indiana University Indianapolis · Indiana University School of Medicine
Affiliation as printed
Indiana Center for Musculoskeletal Health , Department of Medicine, , Indianapolis, IN 46202 , United States
Indiana University School of Medicine , Department of Medicine, , Indianapolis, IN 46202 , United States
Indiana Center for Musculoskeletal Health, Department of Medicine, Indiana University School of Medicine, Indianapolis, IN 46202, United States
-
Massachusetts General Hospital
Affiliation as printed
Endocrine Unit, Massachusetts General Hospital , Boston, MA 02114 , United States
-
Affiliation as printed
Division of Endocrinology, Hospital for Special Surgery , New York, NY 10021 , United States
Metabolic Bone Service, Hospital for Special Surgery , New York, NY 10021 , United States
Research Division, Hospital for Special Surgery , New York, NY 10021 , United States
-
Matthew Blake Greenblatt corresponding
Hospital for Special Surgery · Weill Cornell Medicine
Affiliation as printed
Department of Pathology and Laboratory Medicine, Weill Cornell Medical College , New York, NY 10065 , United States
Research Division, Hospital for Special Surgery , New York, NY 10021 , United States
Cited by 24 stored of 24
No patents citing this paper on Lens.org (checked 2026-10-11).