Human vs AI: A Study on Grammatical Error Recognition

Thursday - January 11 2024, 21:24 UTC - 10 months ago

A study conducted by UAB and URV researchers and published in PNAS showed that humans are significantly better at recognizing grammatical errors in a sentence compared to large language models. The study compared the skills of humans and the three best large language models currently available and highlighted the need for improving language models to narrow the gap between humans and AI.

Language is one of the main features that differentiates human beings from other species. Where it comes from, how it is learned and why people have been able to develop such a complex communication system has raised many questions for linguists and researchers from a wide variety of research fields. In recent years, considerable progress has been made in trying to teach computers language, and this has led to the emergence of so-called large language models, technologies trained with huge amounts of data that are the basis of some artificial intelligence (AI) applications: for example, search engines, machine translators or audio-to-text converters. However, a study conducted by a research team led by the URV with the participation of Humboldt-Universitat de Berlin, the Universitat Autònoma de Barcelona (UAB) and the Catalan Institute of Research and Advanced Studies (ICREA) has shown that there is still a significant gap between humans and AI when it comes to language skills. The study, published in the Proceedings of the National Academy of Sciences (PNAS) journal, compared the skills of humans and the three best large language models currently available: two based on GPT3, and one (ChatGPT) based on GP3.5.

The participants were given a task that was straightforward for people: they were asked to identify on the spot whether a wide variety of sentences were grammatically well-formed in their native language. Both the humans and the language models were asked a very simple question: "Is this sentence grammatically correct?" The results showed that while humans were able to correctly identify grammatical errors, the large language models gave a significant number of incorrect answers. In fact, they were found to adopt a default strategy of answering "yes" most of the time, regardless of whether the answer was correct or not. This is surprising, considering that these systems are trained on the basis of what is grammatically correct or not in a language. Human evaluators explicitly train these models about the grammaticality of different sentence constructions, providing them with examples of both correct and incorrect sentences and giving them feedback on their answers. This type of instruction is a fundamental part of their "training." However, this is not the case for humans, who do not have access to "negative evidence" about what is not grammatically correct in the language being spoken. The study highlights this double mismatch between humans and AI, and brings to light the unique ability of humans to recognize and understand grammatical errors in a sentence.

In conclusion, the study reveals that although artificial intelligence has made great strides in the field of language, humans still possess a unique set of skills that enable them to recognize and understand grammatical errors in a sentence. This highlights the need for continued research and improvement in language models in order to narrow the gap between humans and AI.