A recent study published on February 8, 2023, in JMIR Medical Education, a leading open-access journal on digital medical education, has evaluated the potential of ChatGPT, a natural language processing model, as a tool for medical education. The study, conducted by researchers from Yale University School of Medicine's Section for Biomedical Informatics and Data Science and University College Dublin, aimed to test the performance of ChatGPT and previous-generation large language models on the medical question-answering problem as part of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams.
The researchers tested the models on study aids commonly used by medical students, including multiple-choice questions and the National Board of Medical Examiners (NBME) test sample questions. ChatGPT outperformed previous-generation models and was capable of correctly answering up to over 60% of questions, which is comparable to a passing score for a third-year medical student.
The authors of the study believe that their results make a compelling case for the potential use of ChatGPT as an interactive medical education tool, as it provides users with contextually interpretable and narratively coherent translation of medical knowledge along with its answers. Conrad W Safranek, one of the medical students involved in the project, describes his use of ChatGPT as a study aid. Upon reflection, he found value in using the tool to unearth context relevant to the question, which supported his ability to recall external information and make logical connections from medical courses as expected by the question.
The corresponding author, David Chartash, PhD, from Yale University School of Medicine, remarked that "exposure to the fundamentals of novel technology in pre-clinical years that may shape their practice (such as with dialogic AI) will support their ability to understand the technology-augmented clinical practice they will inherit when they graduate."
Given the interest this research has generated among medical educators and researchers, JMIR Medical Education has released a call for papers for its upcoming theme issue and e-collection titled "ChatGPT and Generative Language Models in Medical Education." This special issue aims to explore the potential of emerging technologies like ChatGPT and similar generative language models or AI applications in medical education, including their use in teaching and learning, clinical decision-making, and patient care.
ChatGPT, the first in a new line of language models, has drawn considerable attention since its prototype was released on November 30, 2022, with users sharing their impressions of the chatbot all across the globe. The potential for ChatGPT to revolutionize medical education is significant, and this study marks a significant advancement in natural language processing models for medical question answering that could have a profound impact on the future learning environment for medical students.
Click here to read the original press release.
