Students may be able to use ChatGPT to help with their university assignments, but the chatbot can lack critical analysis skills Angel Garcia/Bloomberg via Getty Images
It’s possible that ChatGPT outperforms human students on exams in about a fourth of all college-level programmes. This is true, however, only for simple queries that may be answered by recalling information rather than analysing it critically.
Yasir Zaki and his group at New York University Abu Dhabi in the United Arab Emirates reached out to their peers in other divisions, requesting for test questions from a variety of disciplines offered at the school.
The answers offered by these friends were those of actual students. The queries were fed into the AI chatbot ChatGPT, which generated its own answers.
Then, an evaluation group was given access to both sets of answers. Neither the responses’ origins nor the grading’s ultimate goal were communicated to the graders, according to Zaki.
ChatGPT’s answers were considered on par with or better than those of students in 9 out of 32 courses. Sometimes its solutions were vastly superior. When asked questions from a course titled “Introduction to Public Policy,” for instance, it scored nearly twice as high as the average student.
According to Zaki, “ChatGPT did very well on questions that required information recall but very poorly on questions that required critical analysis.”
The findings, according to Thomas Lancaster of Imperial College London, point to a problem with the current system of determining academic standards at universities. Instead than relying on ChatGPT, teachers can challenge their students’ ability to think critically. As the author puts it, “if [better answers are] possible [with ChatGPT], it suggests that there are flaws in the assessment design.”
Lancaster further claims that many of the cheatable assessments on ChatGPT may have been cheated on using preexisting contract cheating services. Professional essay writers are hired by students, who may or may not complete their own critical analysis.
Educators and students in the United Kingdom, the United States, India, Japan, and Brazil were asked their thoughts on ChatGPT in a separate study conducted by Zaki and his team. Students across all regions were more positive about the chatbot’s potential utility than their academic counterparts.
FAQs
1. How does ChatGPT compare to human students in university courses?
ChatGPT performs as well as or even better than students in about a fourth of college-level courses, particularly in answering questions that involve recalling information.
2. What type of questions does ChatGPT excel at?
ChatGPT excels in answering questions that require information recall rather than critical analysis. It provides accurate responses for straightforward queries.
3. What was the methodology of the study comparing ChatGPT’s performance?
Researchers collected test questions from various university disciplines and had ChatGPT generate answers. An evaluation group compared ChatGPT’s answers to those of actual students without knowing their origins.
4. In which courses did ChatGPT perform well?
ChatGPT’s answers matched or surpassed student responses in 9 out of 32 courses. For instance, it significantly outperformed students in the course “Introduction to Public Policy.”
5. What are the implications of ChatGPT’s performance for academic standards?
The study highlights potential issues with the current assessment system in universities. It suggests that teachers should focus on fostering critical thinking skills, as ChatGPT excels in recall-based questions but struggles with critical analysis.
