As AI text generation becomes commonplace, blending this tool with learning goals in academia is a hot topic. While AI text generators can help improve writing, they also create the temptation for students to take short cuts. In the context of academia, AI-generated texts could be used to cheat on university exams, theses, or school homework.

Over-reliance on AI tools does not do students any favours. Although they accomplish tasks more easily in less time, they deprive themselves of experience to conduct research, differentiate a good source from a bad source, imbibe information, and create completely original work.

Teachers have the responsibility to guide students to use AI with integrity. For example, making it okay to use an AI text generator for inspiration or guidance, but ensuring that the final work reflects their own research and words. This adds another layer of evaluation for teachers, distinguishing AI-generated text from that written by a human. In this article, we discuss the AI detector tools available to identify if learners have used AI.

A regular plagiarism scanner does not help

A normal plagiarism scanner determines the uniqueness of texts. It simply checks whether the text already appears elsewhere such as on websites and in books. However, Open AI's ChatGPT - the most popular AI chatbot right now - generates unique texts that have never been formulated like this before. To make it easier to distinguish between text written by a human and an AI, OpenAI launched a tool called AI classifier in January this year – and took it offline again in July.

AI classifier was not fully reliable

The classifier was a language model that predicted how likely a text was written by AI. It was trained on a dataset of pairs of human-written text and AI-written text on the same topic. Each text was divided into a prompt and a response. On these prompts, OpenAI generated responses from a variety of different language models.

The classifier was unreliable for short texts (below 1000 characters) and was recommended only for English texts. In evaluations of English texts, the classifier correctly identified 26% of AI-written text as “likely AI-written”. It incorrectly labelled human-written text as AI-written 26% of the time. Increasing the length of the input text was found to increase the reliability of the tool. AI-written texts could, however, be edited to evade the classifier. Thus, it was impossible to reliably detect all AI-written text.

Alternatives to AI classifier 

There are other classifiers that predict whether a text was written by AI. GPTZero, which has more than a million users, uses two indicators, "perplexity" and "burstiness", to evaluate the complexity of text. If the text is perplexing to GPTZero, then it is likely to have been conceived by a human brain. But if the text appears familiar to the bot, which has been trained on a large and diverse volume of human-written and AI-generated text, it is likely that AI is behind it.

“Burstiness” refers to variations in the length of sentences. Humans tend to create sentences of varying lengths and complexities, while AI tends to create more uniform sentences. GPTZero uses this rationale to inform its predictions. But it is not perfect. In an NPR article, the tool's creator admits it "isn't foolproof" and efforts are on to improve its accuracy.

The AI detector, Giant Language Model Test Room (GLTR), is a collaboration between Harvard University and the MIT-IBM Watson AI Lab. It is based on the fact that AI text generators consider statistical patterns in written text rather than the actual meaning of words and sentences. GLTR identifies whether the words in the text read as if they were written by an AI rather than a human.

When text written by OpenAI's algorithm was put through the tool, it was found to have a lot of predictability. GLTR also outperformed Harvard students in spotting authentic versus AI-created text.

Yet another example is DetectGPT, currently a proof-of-concept (and paper) by scholars at Stanford University. It only detects text created by GPT-2, making it unreliable for other language models. The idea for the AI detector came about when its makers attempted to understand how much, on average, large language models (LLMs) liked human-generated text and their own (LLM-generated) text. LLMs tend to demonstrate a bias for texts that are any slight rephrasing of their own outputs. By contrast, even when an LLM "likes" a piece of human-written text and gives it a high-probability rating, it tends to score any slightly modified version of that text less than the original.

DetectGPT's approach worked better in discerning the correct source of text. In initial experiments, the tool successfully classified human- versus LLM-generated text 95% of the time.

Different approaches to the detection of AI-generated texts

From the discussion of these tools, it is evident that there are different ways to detect AI-generated text. Training a language model with both human and AI-generated texts, the approach originally used by OpenAI, requires huge amounts of data and may not be feasible for all developers.

An alternative to training a new model is to use an existing model to detect its output and score the text accordingly, in the style of DetectGPT. Watermarking is another approach proposed to expose AI-generated text. It involves burying secret patterns in LLMs that computers can detect but are invisible to the human eye.

AI content detectors can fall short of their job

Students can outsmart AI content detectors. All they have to do is edit the AI-generated text and the results will skew in their favour.

Watermarking is a valid solution, but watermarks have to be embedded in chatbots from the start. One of the promising applications of AI-generated text is integrating it into products as a spell checker in a word processor or as a tool to help people write emails. This is acceptable and does not constitute cheating. But as watermarking can generate false positives, AI providers may think twice about using it to prevent the risk of unfair accusations and needless controversy.

ChatGPT and other AI text generators need a continuous supply of new training data to be able to create different versions of similar queries. Constraints in training data and programming mean that the texts generated by the tools will eventually start resembling each other, creating complexities for AI detection tools. Drawing judicious conclusions may necessitate the use of not one, but two, or three different detectors. 

Students who use ChatGPT or other AI text generators as a substitute for their education forget the purpose of education. Further research is needed to tackle the challenges of this new technology and its impact on the educational system.