A study on ChatGPT and learning retracted almost a year after publication

A study claiming to demonstrate the positive impact of ChatGPT on school learning has been retracted nearly a year after its publication. Springer Nature, the publisher of the work, cited "discrepancies" in the analysis and lack of confidence in the conclusions. The retraction occurred after the paper had accumulated hundreds of citations and received widespread attention on social media.

Quick Answer

The retracted study claimed that ChatGPT significantly improved learning performance, learning perception, and higher-order thinking of students. Springer Nature retracted the paper due to discrepancies in the analysis and lack of confidence in the conclusions. The work had received 504 citations and widespread online attention before the retraction.

Technical and Methodological Analysis

The paper, published on May 6, 2025, in the journal Humanities & Social Sciences Communications, was a meta-analysis that analyzed the results of 51 previous studies. The authors had calculated the effect of ChatGPT on learning performance, learning perception, and higher-order thinking, comparing experimental groups using AI with control groups that did not.

According to the authors, the analysis showed a "large positive impact" of ChatGPT on learning performance, a "moderately positive impact" on learning perception, and a "positive effect" on higher-order thinking. However, Ben Williamson, senior lecturer at the Centre for Research in Digital Education at the University of Edinburgh, criticized the study's methodology.

Criticisms of Methodology and Data Quality

Williamson highlighted that the study synthesized "few low-quality studies" or mixed results from research with methods, populations, and samples too different to be accurately compared. "It seemed like a paper that should not have been published from the start," he stated.

Another critical point concerns the timing of the publication. ChatGPT was released by OpenAI only in November 2022, and Williamson observed that it was unlikely that dozens of high-quality studies on ChatGPT and learning could be conducted, reviewed, and published in less than two and a half years.

Impact and Spread of the Study

Despite the retraction, the study had a significant impact. Before the retraction, the paper had received 504 citations, 262 of which were from articles published in Springer Nature's peer-reviewed journals. Additionally, it attracted nearly half a million readers and widespread online attention, ranking in the 99th percentile for attention score among journal articles.

Implications for AI and Education Research

This case raises important questions about the quality and validity of studies assessing the impact of artificial intelligence technologies on learning. The rapid spread and citation of works that do not pass a rigorous methodological examination can lead to incorrect conclusions and negatively influence educational policies.

For research institutions and academic journals, this episode underscores the importance of adopting more rigorous review procedures, especially when dealing with emerging technologies like generative AI. The need for well-designed studies based on solid data is crucial to fully understand the role of these technologies in the field of education.

Future Considerations

While the retraction of the study represents a step back in research on the use of ChatGPT in education, it also offers the opportunity to improve research standards in this field. Scholars should focus on more robust methodologies and analyses that take into account the various variables involved in the use of artificial intelligence tools in learning.

The Context of AI and Education Research

This episode fits into a broader context of growing interest in the application of artificial intelligence technologies in the field of education. In recent years, numerous studies have explored the potential of tools like ChatGPT to improve learning, teaching personalization, and administrative efficiency. However, the speed at which these technologies evolve has made it difficult for the scientific community to conduct rigorous and long-term research.

The retracted meta-analysis attempted to fill this gap but failed to ensure the quality and consistency of the data analyzed. This case highlights the need for more cautious and methodologically sound approaches when evaluating emerging technologies. The scientific community must avoid being influenced by the speed at which these technologies spread and the enthusiasm they can generate on social media.

Consequences for Educational Policies

One of the most concerning aspects of this retraction is the impact the study might have had on educational policies. Many institutions and policymakers could have used the study's results to justify investments in artificial intelligence technologies or to modify educational programs. The spread of incorrect information can lead to ill-informed decisions, with negative consequences for students and the educational system as a whole.

To prevent similar situations from recurring, it is essential that educational institutions adopt an approach based on solid and rigorous evidence. This means not only waiting for research to be more consolidated but also promoting a culture of critical skepticism and independent verification of results.

The Challenges of Peer Review

The case of the retracted study also raises important questions about peer review and the scientific publication process. The speed with which the study was published and its subsequent spread on social media suggest that the review system may not be adequate to handle the growing amount of research on emerging technologies. Academic journals must address these challenges by implementing more rigorous and transparent review procedures.

In particular, it is crucial that reviewers are experts not only in the field of education but also in artificial intelligence. This will ensure that studies are accurately evaluated and that any methodological discrepancies are identified promptly. Additionally, journals should consider establishing a continuous review process that allows for updating and correcting published studies based on new evidence.

The Future of AI and Education Research

Despite the difficulties, the retraction of the study represents an opportunity to improve the quality of research in this field. Scholars should focus on more robust methodologies, such as longitudinal studies and controlled experiments, that can provide a deeper understanding of the impact of AI on learning. Additionally, it is essential to promote collaboration between researchers, educational institutions, and technology developers to ensure that research is based on solid and relevant data.

The case of the retracted study on ChatGPT and learning underscores the importance of a critical and rigorous approach to scientific research. The educational and scientific communities must work together to ensure that artificial intelligence technologies are accurately evaluated and that decisions based on these evaluations are informed and well-founded.

Editorial Note and Disclaimer

The guides and content published on GoYou are the result of independent research and analysis activities, for informational, educational, and in-depth purposes.

GoYou does not constitute a journalistic publication or an editorial product pursuant to Law No. 62/2001 and does not provide real-time information.

The GoYou project does not provide professional, technical, legal, or financial advice and disclaims all responsibility for the improper use of the information published.

In the Crypto sector, every investment involves risks: readers are invited to always inform themselves autonomously before making any decision.