Chatbots Do Not Perform Well for Misinformation-Prone Health Topics

Follow Drugslib on

By Lori Solomon HealthDay Reporter

Medically reviewed by Drugs.com

via HealthDay

FRIDAY, April 17, 2026 -- Chatbots perform poorly when answering questions about misinformation-prone health topics, according to a study published online April 14 in BMJ Open.

Nicholas B. Tiller, Ph.D., from Harbor-UCLA Medical Center in Torrance, California, and colleagues audited chatbot responses for health-related questions prone to misinformation. Ten questions from five categories (cancer, vaccines, stem cells, nutrition, and athletic performance) were used as prompts in five popular chatbots: Gemini (Google), DeepSeek (High-Flyer), Meta AI (Meta), ChatGPT (OpenAI), and Grok (xAI) in February 2025, with two experts rating responses.

The researchers found that nearly half (49.6 percent) of responses were problematic (30 percent somewhat problematic and 19.6 percent highly problematic). Quality of responses was similar among chatbots (P = 0.566), although Grok generated significantly more highly problematic responses than would be expected under a random distribution (z-score, +2.07). For vaccines (mean z-score, –2.57) and cancer (–2.12), performance was strongest, while it was weakest in stem cells (+1.25), athletic performance (+3.74), and nutrition (+4.35). The quality of references was poor, with a median completeness score of 40 percent. No chatbot produced a fully accurate reference list due to hallucinations and fabricated citations. Readability of responses was graded as "difficult," equivalent to college sophomore–senior level.

"By default, chatbots do not access real-time data but instead generate outputs by inferring statistical patterns from their training data and predicting likely word sequences," the authors write. "They do not reason or weigh evidence, nor are they able to make ethical or value-based judgments. This behavioral limitation means that chatbots can reproduce authoritative-sounding but potentially flawed responses."

Abstract/Full Text

Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.

Source: HealthDay

Posted : 2026-04-18 01:55

Read more

Disclaimer

Every effort has been made to ensure that the information provided by Drugslib.com is accurate, up-to-date, and complete, but no guarantee is made to that effect. Drug information contained herein may be time sensitive. Drugslib.com information has been compiled for use by healthcare practitioners and consumers in the United States and therefore Drugslib.com does not warrant that uses outside of the United States are appropriate, unless specifically indicated otherwise. Drugslib.com's drug information does not endorse drugs, diagnose patients or recommend therapy. Drugslib.com's drug information is an informational resource designed to assist licensed healthcare practitioners in caring for their patients and/or to serve consumers viewing this service as a supplement to, and not a substitute for, the expertise, skill, knowledge and judgment of healthcare practitioners.

The absence of a warning for a given drug or drug combination in no way should be construed to indicate that the drug or drug combination is safe, effective or appropriate for any given patient. Drugslib.com does not assume any responsibility for any aspect of healthcare administered with the aid of information Drugslib.com provides. The information contained herein is not intended to cover all possible uses, directions, precautions, warnings, drug interactions, allergic reactions, or adverse effects. If you have questions about the drugs you are taking, check with your doctor, nurse or pharmacist.

Popular Keywords