Do Smart Machines Make Smarter Students? Rethinking Assessments in the Age of AI
by Gary L. Beck Dallaghan, Ph.D. | October 29, 2024
Article Citation: Hersh W, Fultz Hollis K. Results and implications for generative AI in a large introductory biomedical and health informatics course. NPJ Digit Med. 2024 Sep 13;7(1):247. DOI: 10.1038/s41746-024-01251-0
What is this article about?
Large language models performed at the 50th to 75th percentile of students in this study. The authors explore performance of multiple generative artificial intelligence (AI) systems, comparing their performance to students’ on summative assessments in a biomedical and health informatics course. It focuses on the accuracy of AI in answering multiple-choice and final exam short answer questions in a course taken by graduate, continuing education, and medical students. Based on their findings, the authors raise questions about the future of student assessment in knowledge-based educational settings.
Why should you read the article?
This article reports on an investigation of how large language model generative AI systems perform on academic assessments. This is a topic highly relevant to educators given the increasing utilization, both formally and informally, of AI tools in educational and professional domains. The results of this study highlight strengths and limitations of AI in supporting or undermining learning. The authors raise important considerations for assessment integrity in the era of readily accessible generative AI and possible mitigating strategies, making it essential reading for those in education, technology, and healthcare.
How can you use this article?
The findings of this study prompt educators to rethink assessment strategies, for instance by developing more complex questions. Writing questions of this caliber are referred to as “Google proof”, which means that if you can question Google and get the answer it is not complex enough. Additionally, educators should consider adjusting assessment methods to account for AI use, such as ending open-book or remote testing formats. The age of generative AI is here. Medical educators therefore need to consider the ethical use of AI and its role in learning, as well as prepare students for using AI both in learning and in healthcare.
Review Author: Gary L. Beck Dallaghan, Ph.D., Assistant Dean for Accreditation, Carle Illinois College of Medicine, Urbana, IL. Organization: Council on Medical Student Education in Pediatrics