By Dan Primack, author of Axios Pro Rata
ChatGPT, the generative AI juggernaut, is getting a lot smarter when it comes to health care.
Why it matters: A lot of clinical diagnoses and decisions could someday be made by machines, rather than by human doctors.
Driving the news: ChatGPT recently passed all three parts of the U.S. Medical Licensing Examination, although just barely, as part of a recent research experiment.
- As the researchers note, second-year medical students often spend hundreds of hours preparing for Part 1, while Part 3 usually is taken by medical school graduates.
Zoom in: Ansible Health, a Silicon Valley startup focused on treating COPD, had been researching various AI and machine learning tools to improve its care.
- “There was so much excitement in the tech world when ChatGPT came out, so we wanted to see if it was just hype or useful,” explains Jack Po, Ansible’s CEO and a former Google product manager.
- “As we started doing validation we were pretty amazed at the results. Not only at what it was getting right, but at how it was explaining itself.”
- Po and several others then decided to have ChatGPT take the USMLE, first ensuring that “none of the answers, explanations or related content were indexed on Google.” They then published their results, which currently are undergoing peer review.
The big surprise was that ChatGPT could perform so well without ever having been trained on a medical dataset.
- One caveat is that researchers excluded a set of “indeterminate” answers, as it appears that ChatGPT was programmed to avoid providing what could be construed as medical advice.
- “Those answers were so general that it was hard to say if they were right or wrong,” explains paper co-author Morgan Cheatham, an investor with Bessemer Venture Partners and current medical school student at Brown University.
Between the lines: Generative AI remains in the early innings, so for now it’ll augment medical work rather than replace it.
- Ansible, for example, is using ChatGPT to help explain certain concepts to patients, after review by a trained professional.
What’s next: Over time, perhaps it could be applied to wellness checks and other general practitioner tasks.
- Once the technology moves beyond just text, it could incorporate data inputs like vocal tone, body language and facial expressions.
- One benefit would be the immediate incorporation of a patient’s medical records. Sometimes physicians only have moments to scan a lifetime of charts.
Reality check: Don’t expect a machine to autonomously diagnose patients anytime soon. AI models like ChatGPT sometimes make confident assertions that turn out to be false — which could prove dangerous in medical applications.
What they’re saying: “I think we’re in the middle of a 20-year arc, kind of like what we already saw with finance,” says Vijay Pande, a health care investor with Andreessen Horowitz and adjunct professor of bioengineering at Stanford University.
- “In 2000, it was insane to think that a computer could beat a master trader on Wall Street. Today, it’s insane to think that master trader could beat a computer.”
The bottom line: Plenty of people rely on “Dr. Google” for their medical information needs. In the future, they may turn to “Dr. ChatGPT.”
Editor’s note: This story has been corrected to reflect that Part 1 of the USMLE is usually taken after the second year of medical school.