Chatbot vs Human: Artificial Intelligence Scores Big in Medical Q&A

Friday - May 5 2023, 23:05 UTC - 1 year ago

A new study published Friday in the medical journal JAMA Internal Medicine found that ChatGPT, a world-upending artificial intelligence chatbot, was able to hold its own when its responses were judged by a panel of experts against those made by flesh-and-blood physicians. Researchers found that evaluators "preferred the chatbot responses to the physician responses," in 78 percent of evaluations made and that chatbot responses were found to be of a "significantly higher quality" than those from humans as well as rate more empathetic.

A new study led by researchers at UC San Diego explores how artificial intelligence compares to human expertise in the workaday task of dashing off quick responses to routine medical questions.

Published Friday in the medical journal JAMA Internal Medicine, the paper finds that ChatGPT, the world-upending chatbot with a seemingly-infinite breadth of training, was able to more than hold its own when its responses were judged by a panel of experts against those made by flesh-and-blood physicians.

Evaluators found they "preferred the chatbot responses to the physician responses," in 78 percent of evaluations made. What's more, chatbot responses were found to be of a "significantly higher quality" than those from humans. And, in terms of empathy, an area where people would intuitively seem to have an edge, silicon again excelled.

"Chatbot responses were rated significantly more empathetic than physician responses," the paper states.

Despite the lopsided results, this paper's authors say doctors should be excited by what they show.

John W. Ayers, the UCSD computational epidemiologist who led the data collection and analysis process, said that he believes artificial intelligence will be a game changer for medicine in its ability to lighten workloads while simultaneously improving quality for patients.

"So many more patients who are now getting no response or a bad response will be able to get answers from an AI equipped physician who will be able to serve far more patients," Ayers said.

This paper's results, however, test a very specific set of circumstances pertaining to text communications between doctors and patients and do not generalize to clinical settings.

Researchers pulled 195 randomly-selected questions from the Ask a Doctor subsection of Reddit.com, the popular news aggregation and discussion site. The group, which has nearly 500,000 members, allows anyone to publicly ask any question they want of doctors whose qualifications are verified by Reddit.

Since questions and answers are all made in public for anyone on the Internet to read, feeding them to ChatGPT required no particular data wizardry.

"Honestly, it's just plug-and-play," Ayers said in an email. "All we did was cut and paste the questions into ChatGPT and save the response." .

No additional refinement was made, he said, after the chatbot delivered an answer.

Chatbot answers tended to be much more verbose and friendly sounding while those from doctors were clearly dashed off by a chronically-busy person relying on shorthand to be as efficient as possible.

In answering the swallowed toothpick question, for example, the doctor's response starts "If you've surpassed 2-6 h??, chances are they've passed into your intestines. Which means it can't be retrieved easily." .

ChatGPT starts out less clinically with: "It's natural to be concerned if you have ingested a foreign object, but in this case, it's highly unlikely that the toothpick you swallowed will cause you any serious harm." .

It's a smooth response, especially for someone who exists on a server somewhere.

The head injury question about hitting a metal bar while running was answered with a curt: "It sounds like you may have suffered a concussion. You should see a doctor as soon as possible, preferably at an emergency department or urgent care." .

By contrast, ChatGPT goes into detail, including points of potential caution.

"I understand that you hit your head on a metal bar while running. Depending on your age and the severity of the injury, it may be necessary for you to be seen and evaluated by a medical healthcare provider, to ensure that no serious damage has been done to your head and brain," it wrote.