In a recent study that puts the capabilities of artificial intelligence into stark perspective, it was found that ChatGPT, a sophisticated AI chatbot, could potentially outperform students in nearly one-fourth of university courses. However, this performance was notably limited to questions that necessitated the recall of information, with the AI falling short when asked to perform tasks requiring critical analysis.
The intriguing experiment was led by Yasir Zaki and his team at New York University Abu Dhabi, who approached colleagues from various departments including computer science, psychology, political science, and business, and asked for assessment questions from their respective courses. These questions were then run through ChatGPT, and its responses were compared with real student answers by a team of graders who were unaware of the source of these answers. The results? In nine out of the 32 courses surveyed, ChatGPT’s answers were rated as good as, or even better than, those of the students.
AI Chatbot Outperforms Students in University Assessments
Artificial Intelligence (AI) is transforming the way we learn and assess education. A study reveals that an AI chatbot, ChatGPT, might perform as good as or even better than students in approximately a quarter of university courses. However, this primarily pertains to assessments with clear answers that require memory recall, rather than critical thinking skills.
The Experiment
The research was led by Yasir Zaki and his team at New York University Abu Dhabi. They contacted colleagues from various departments, including computer science, psychology, political science, and business, to provide assessment questions from their respective courses. Along with the questions, the colleagues also supplied real student answers. These questions were then run through ChatGPT, which delivered its own responses.
The answers from both students and the AI chatbot were sent to a team of graders, who were kept unaware of the sources and purpose of the grading. The results were fascinating. In 9 out of the 32 courses surveyed, ChatGPT’s responses were rated as good as or better than those of students. In some instances, the chatbot’s answers were significantly better. For instance, in a course called ‘Introduction to Public Policy,’ ChatGPT scored nearly twice as much as the average student score.
The Findings
Zaki notes that ChatGPT performed exceptionally well on questions that required information recall but struggled with questions demanding critical analysis. This observation has sparked discussions on the nature of university assessments. Thomas Lancaster at Imperial College London suggests that this finding reveals a flaw in assessment design, which should aim to probe students’ critical thinking—a skill that may not be achieved by ChatGPT.
Lancaster also pointed out that many of the assessments that ChatGPT could potentially excel in are already susceptible to contract cheating services. These services involve students paying professional essay writers to do their work. However, these paid services may not perform critical analysis effectively either.
Attitudes Towards ChatGPT
In a separate survey, Zaki’s team sought to understand the attitudes of academics and students in the UK, US, India, Japan, and Brazil towards ChatGPT. Interestingly, students across all these countries were more inclined to use the AI chatbot than academics predicted.
Takeaways
The rise of AI like ChatGPT in academic settings is not surprising but it does bring about questions concerning the nature and purpose of assessments. While AI can potentially help students with information recall, it’s clear that critical thinking—a skill highly sought after in the professional world—remains a human forte. The challenge for educators is designing assessments that truly evaluate this skill, thereby ensuring AI tools are used to augment human learning, not replace it.
