How to Detect AI Generated Text Accurately

Introduction

Learning how to detect AI generated text accurately has become a practical skill for teachers, editors, hiring managers, and anyone who reads the open web. The stakes are clear because OpenAI withdrew its own classifier after it caught only 26 percent of AI-written text while wrongly flagging 9 percent of human writing. That result surprised people who assumed a detector works like a lie detector for prose. In reality every detector outputs a probability, and every probability hides an error rate that shifts with the text, the author, and the model that produced it. This guide walks through the linguistic signals and the statistics behind perplexity and burstiness. It also supplies code you can run yourself and a repeatable workflow that treats a score as evidence instead of a verdict. You will also see where detectors fail, who they fail most often, and what watermarking might change. By the end you will be able to reach a defensible conclusion about any document and explain exactly how confident you are.

Quick Answers on How to Detect AI Generated Text Accurately

What is the most accurate way to detect AI generated text?

Combine several weak signals: linguistic patterns, two independent detector scores, and process evidence such as drafts and version history. No single tool is reliable enough alone, so report a probability, not a verdict.

Can AI detectors be trusted to prove someone used AI?

No. Detectors estimate likelihood and make mistakes in both directions. Vanderbilt calculated that a 1 percent false positive rate would wrongly flag about 750 of its 75,000 yearly papers, so scores alone cannot justify accusations.

Does perplexity and burstiness analysis detect ChatGPT writing?

It helps but is not decisive. Low perplexity and uniform sentence rhythm are common in machine text, yet formal human writing and non-native English also score low, which causes false positives.

Key Takeaways

• Treat every detector score as a probability with a known error rate, then combine it with linguistic checks and process evidence before reaching a conclusion.
• Peer-reviewed work found detectors flagged an average of 61.22 percent of non-native English TOEFL essays as AI, so fairness risk is highest for second-language writers.
• Perplexity, burstiness, repeated phrasing, and vague significance claims are useful clues that you can measure with a short Python script, but none proves authorship alone.
• Paraphrasing and humanizer tools defeat most detectors, while watermarks such as SynthID may provide stronger provenance in the future but only for text from cooperating models.

Understanding How to Detect AI Generated Text Accurately

How to detect AI generated text accurately means combining linguistic analysis, statistical detector scores, and process evidence, then estimating the probability of machine authorship with stated uncertainty rather than declaring certainty from any single tool or signal.

How AI Text Detectors Work Under the Hood

Every AI text detector falls into one of four families, and knowing which family a tool belongs to tells you what it can and cannot see. Supervised classifiers are neural networks trained on large collections of labeled human and machine writing, then asked to guess which label fits a new passage. Zero-shot statistical methods skip the training step and instead score how probable the passage looks to a reference language model. Watermark detectors check for a hidden statistical pattern that the generating model inserted on purpose, and stylometric tools measure habits such as sentence length, vocabulary range, and punctuation. Most consumer tools you meet online combine at least two of these ideas behind a single friendly score.

The statistical family deserves a closer look because it powers most free tools. Language models assign a probability to each next word, and text sampled from a model tends to sit in regions where those probabilities are high and smooth. The DetectGPT paper formalized this idea by showing that model-written passages sit at points of negative curvature in the log probability function. That insight lifted detection AUROC from 0.81 to 0.95 on GPT-NeoX fake news. Binoculars later refined the approach by contrasting two closely related models. Its authors report that it catches over 90 percent of generated samples at a 0.01 percent false positive rate without any training data. These are laboratory numbers on curated datasets, so they describe the ceiling and not the everyday experience of a teacher pasting a student essay into a website.

Commercial products blend several of these signals into one calibrated score, and the vendor decides where the threshold sits between catching more AI text and wrongly accusing more humans. Turnitin, for example, splits a document into overlapping segments, scores each one, and aggregates the results into a document percentage plus highlighted sentences. GPTZero says its upgraded model uses a deep learning architecture with seven indicators, which is a long way from the simple perplexity and burstiness meter it launched with. Because the thresholds, training data, and update schedule are proprietary, two tools can disagree sharply on the same essay and both be behaving as designed. That is why serious users treat a vendor score as one input and ask for the false positive rate at the exact threshold being applied.

Why Accuracy Is Harder Than Vendors Admit

Accuracy sounds like a single number, yet a detector has two separate error rates that trade off against each other. The false positive rate is the share of genuine human documents wrongly labeled as AI. The false negative rate is the share of machine documents that slip through as human. Lowering the threshold catches more AI text but flags more people. Raising it protects people but lets more machine text pass. Marketing pages usually advertise the best-looking pair of numbers, often measured on clean, unedited output from one popular model, which is a friendly setting that real-world documents rarely match.

The base rate problem is the part most people miss, because even a very low false positive rate produces many wrong flags when the number of honest documents is large. Vanderbilt University did the arithmetic in August 2023 when it disabled Turnitin’s AI detector. It noted that a 1 percent rate applied to the 75,000 papers submitted in 2022 implied roughly 750 students falsely flagged. Now bring prevalence into the picture, meaning the true share of documents that contain AI text. If only 2 percent of submissions contain AI text, a detector with 80 percent sensitivity and a 1 percent false positive rate is right about 62 percent of the time. At 10 percent prevalence the same tool is right about 90 percent of the time, so the identical score means very different things in different classrooms.

Real documents also differ from benchmark documents in ways that quietly degrade performance. Students edit machine drafts, mix their own paragraphs with generated ones, run text through grammar tools, and translate between languages, and each step blurs the statistical fingerprint. Benchmark studies confirm that detectors lose accuracy as soon as conditions drift away from clean, unedited output. Turnitin itself admits that real-world results differed from its laboratory testing and that it cannot fully remove false positives. When the vendor concedes this much, a single percentage on a dashboard cannot carry the weight that some institutions place on it.

Finally, accuracy decays over time because the generators keep changing. A classifier trained on text from one model generation learns that generation’s quirks, and each new model release shifts the distribution away from what the classifier knows. The AIMultiple benchmark illustrates this pattern across several of the most popular chatbots. Its ten tools correctly identified 87 percent of ChatGPT samples but only 54 percent of Gemini samples in a small test of 18 texts. The sample was tiny and the authors report no confidence intervals, so treat the exact figures cautiously and notice the direction instead. Detector performance depends heavily on which generator wrote the text, which you often do not know in advance.

Linguistic Signals That Separate Human and Machine Prose

Beyond the raw statistics, careful readers notice habits of phrasing that language models produce more often than people do. Wikipedia editors working in an effort called AI Cleanup compiled a field guide to these habits. TechCrunch’s summary of the guide highlights stock claims of importance such as “a pivotal moment”. It also highlights vague trailing clauses that end with phrases like “emphasizing the significance” of a topic. The guide also flags promotional tone, where landscapes are always scenic and everything sounds like the script of a television commercial. These patterns are rare in encyclopedic human writing but common across the internet, which is exactly why models trained on that internet reproduce them. Crucially, the editors call automated detectors basically useless and rely on human judgment about habits instead.

The strongest linguistic evidence is a cluster of several independent habits appearing together, because any single habit also appears in plenty of honest human prose. Look for uniform paragraph length, where every paragraph runs four or five sentences of nearly identical size. Look for lists of exactly three items, hedged openings such as “it is important to note,” and conclusions that restate the introduction in fresh words. Watch for abstract nouns doing the work that a specific example should do, so that a paragraph about teamwork never names a team, a date, or a decision. Notice also what is missing: personal anecdote, a surprising fact only an insider would know, and a clear opinion that risks being wrong.

Factual texture is another useful signal because models smooth over details they cannot verify. A human reviewer who checks one claim in each paragraph against the cited source will often find invented quotations, plausible but nonexistent references, or statistics that nobody published. Our guide to detecting LLM hallucinations explains why these fabrications arise and how to test for them systematically. A document whose references fail to resolve is suspicious regardless of any detector score, and this check is difficult for a paraphrasing tool to hide. Remember the reverse as well: a clean set of verifiable citations does not prove human authorship, since a careful person can edit a machine draft into accuracy.

Perplexity and Burstiness Without the Jargon

Building on those signals, perplexity measures how surprised a reference language model is by each word in a passage, averaged over the whole text. GPTZero describes it as how likely an AI model would have chosen the exact same set of words. A sentence that ends predictably after “Hi there, I am an AI” scores low, while an unexpected word scores high. Mathematically it is the exponential of the average negative log probability the model assigned to each token, which means a lower number equals a more predictable text. Models generate by repeatedly choosing likely tokens, so their output tends toward low perplexity, and that is the whole intuition behind the metric. You can read how the calculation is done in practice in the Hugging Face perplexity guide, which also explains the sliding window needed for long documents.

Burstiness captures the rhythm of the writing by asking how much perplexity or sentence length varies from one sentence to the next. People tend to alternate between long, winding sentences and short punchy ones, and they drop in an odd word choice without warning. Language models, especially at default settings, keep a steadier pace, so their burstiness is lower. A simple way to approximate it without any model is the coefficient of variation of sentence length, which is the standard deviation of words per sentence divided by the mean. The how-to section below includes a script that computes this coefficient, and you should expect the ranges for human and machine text to overlap heavily.

Perplexity and burstiness are weak on their own because they measure predictability, and predictability is a property of the text rather than proof of who wrote it. Formal legal prose, scientific abstracts, and the Declaration of Independence are all highly predictable. When Decrypt tested the preamble, ZeroGPT reported 97.93 percent AI-generated while QuillBot and GPTZero judged it human. Text that appears many times in training data gets memorized, which drives perplexity down for reasons that have nothing to do with machine authorship. Non-native writers who use common words and simple structures also score low, which is the mechanism behind the bias discussed later in this guide. Use the metrics to prioritize attention, never to close a case.

There are practical limits to the do-it-yourself version as well. The number you compute depends on which reference model you use, so a perplexity of 40 under one model cannot be compared with a perplexity of 40 under another. GPTZero’s own explainer concedes there is no absolute scale and offers only a rule of thumb that values above 85 lean human. Short passages under roughly 150 words give noisy estimates because a few unusual tokens swing the average. For that reason the scripts in the how-to section below report the numbers alongside a plain warning about length and never convert them into a verdict.

Reading a Detector Score Without Fooling Yourself

Shifting from raw metrics to interpretation, a detector score is a probability under a model and not a measurement of the document. The first discipline is to ask what the number actually claims. A reading of 80 percent AI can mean the tool believes 80 percent of the sentences are machine written. It can also mean the tool is 80 percent confident that the whole text is machine written. Those are very different statements, and vendors use both of them in their dashboards. Turnitin, for instance, reports the share of the document it believes was written by AI. It shows an asterisk when that figure is below 20 percent, because its own testing found more false positives in that range. Always read the vendor documentation before you trust the number on the screen.

Ask three questions of every score: what is the false positive rate at this exact threshold, what kinds of text was it validated on, and how long is the passage. Turnitin states a document-level false positive rate under 1 percent for documents with more than 20 percent AI writing detected. The sentence-level rate is about 4 percent, according to the company’s own product update. It raised the minimum submission length from 150 to 300 words to reduce errors, which shows that short texts are where scores are least stable. If the vendor cannot answer these questions, discount the score heavily. If it can, write the answers down next to the score so that anyone reviewing your reasoning can see what you assumed.

Two scores from two independent tools are more informative than one, but only if the tools are genuinely independent. Many cheap websites wrap the same open-source classifier, so agreement between them adds little. Choose one tool from a different family, such as a supervised commercial classifier plus a zero-shot method like Binoculars, and compare. When they disagree, the disagreement itself is useful information, because it usually means the passage is short, edited, translated, or from a model neither tool has seen. In that case fall back on the linguistic and process evidence rather than averaging the numbers.

Finally, remember that a score never answers the question people actually care about, which is whether this person did something wrong. Using AI for brainstorming, grammar help, or translation may be allowed under a given policy even when it lifts a detector’s output. A high score therefore starts a conversation and does not end one. Reviewers who keep this framing avoid the worst outcomes, which are wrongful accusations built on a threshold and a screenshot. The conversation should include asking the author to explain choices, show drafts, and discuss the sources, as the workflow later in this guide describes.

Comparing Detection Approaches Side by Side

Stepping back from individual metrics, each family of methods trades coverage, cost, and robustness differently, and choosing among them depends on whether you control the text source. Supervised classifiers are fast and convenient, and they can perform well on the model families they saw during training. They degrade when a new model appears and they offer little explanation for a verdict. Zero-shot methods such as Binoculars need two reference models and more computing power, but they generalize better because they rely on a property common to many generators. Neither family is trustworthy on short or heavily edited text.

Watermarking is the only approach that offers a statistical guarantee, but it works only when the generating model cooperated and the text has not been rewritten. Kirchenbauer and colleagues proposed biasing generation toward a pseudorandom green list of tokens so that a detector can later run a significance test. Their watermark paper shows the test needs no access to the model itself. Stylometry and human review complement all of these because they cost little and provide an interpretable explanation that you can show to the author. Our earlier explainer on why detectors remain flawed reaches the same conclusion. The practical answer is therefore a layered one, where cheap linguistic checks come first, one or two detectors come second, and provenance evidence settles the hard cases. The comparison table later in the article sets the approaches side by side.

Putting a Detection Workflow Into Practice

Turning to execution, a workable detection workflow has five stages that move from cheap checks to costly ones, and each stage can end the process early. The stages are intake, a careful human read, quantitative signals, independent detector scores, and finally context gathering with the author. Skipping straight to the detector score is the most common shortcut, and it is also the stage with the least explanatory power. A reviewer who starts with the cheap human read often finds the question already answered. The text may contain a fabricated citation or an obvious copy of a known source. The order matters because early stages cost minutes while the final stage costs a relationship.

Intake is the least glamorous stage and the one that protects you from avoidable errors. Collect the original file rather than a pasted copy, because revision history and metadata in word processors and cloud documents often tell the story faster than any statistic. Record the word count, since passages under roughly 300 words give unstable scores, and note whether the text includes quotations, code, tables, or references that would distort a statistical measure. Capture the assignment prompt, the policy in force, and what kinds of AI assistance were allowed. Without that policy the word “AI generated” has no clear meaning, and a debate about the score will turn into a debate about definitions.

Translate your evidence into one of four levels, and only act on a level that your policy says justifies action. The first level is no concern, where the text reads naturally and no signal stands out. The second is weak concern, where one signal such as low burstiness or a detector flag appears alone, which justifies a friendly question but nothing more. The third is moderate concern, where two independent signals agree, such as stock phrasing plus two detector flags, and this level justifies requesting drafts or a short oral explanation. The fourth is strong concern, which requires process evidence such as a copied chat transcript, a watermark detection, an unexplained jump in style from earlier work, or an admission. Our overview of AI detector tools, risks, and effective use covers how to phrase policies around those levels.

Careful documentation keeps the whole process honest and protects everyone involved. Write the finding as a likelihood statement, for example “moderate concern, two of four signals present, no process evidence,” rather than as a conclusion about guilt. Keep an evidence log listing each tool, its version, the date, the threshold, and the exact score so that a second reviewer can reproduce your reasoning. Ask a colleague to read the evidence blind before any conversation with the author takes place. Re-run the analysis whenever a major new model appears, because a method that worked on last year’s generators may fail on this year’s.

Where Detection Falls Short: Paraphrasing and Humanizers

Looking at failure modes in detail, the most damaging attack is also the simplest, which is to rewrite machine output until the statistical fingerprint disappears. Sadasivan and colleagues tested a recursive paraphrasing attack in which one model rewrites text, then another rewrites the result, over several rounds. They found it substantially lowers detection rates across watermark-based, neural network-based, zero-shot, and retrieval-based detectors while often causing only slight drops in text quality. Their experiments used passages of about 300 tokens, which is close to the length of a typical homework answer. Commercial humanizer tools automate the same idea, and anyone with a browser can use them.

Sampling choices and model updates cause failures even without a deliberate attack. The RAID benchmark covered 11 generative models, 8 domains, 4 decoding strategies, and 11 adversarial attacks, then scored 12 detectors across more than six million generations. Its authors report that current detectors are easily fooled by adversarial attacks, variations in sampling strategy, repetition penalties, and generative models they have never seen. In other words, a detector validated on default settings of one chatbot can collapse when a user raises the temperature or picks a different model. Because you rarely know which model or settings produced a document, you cannot know which part of the benchmark applies to your case.

Hybrid writing is the hardest case of all, because a human who edits a machine draft produces text that is statistically neither one thing nor the other. Consider a student who asks a chatbot for an outline, writes the paragraphs personally, then uses a grammar tool to smooth the result. Sentence-level highlights from a detector will flag some of those sentences and miss others, and each highlight carries its own error rate. Translation round trips add noise too, since text moved from one language to another and back loses much of the original authorial rhythm. The same team also showed that attackers can spoof detectors by inferring hidden signatures, which raises the uncomfortable possibility that someone could make a human’s writing look machine-made on purpose.

Ethics and Fairness for Non-Native Writers

Beyond the question of accuracy, fairness is where detectors cause the most serious harm. A Stanford-led study tested seven widely used detectors on 91 TOEFL essays written by non-native English speakers and found an average false positive rate of 61.22 percent. All seven detectors unanimously labeled 18 of those 91 essays, or 19.78 percent, as AI generated, even though real students wrote every one of them. Essays by native-speaking US eighth graders were classified accurately in the same study. A tool with that pattern does not merely make random errors; it systematically penalizes people for the way they learned English.

The mechanism follows directly from how perplexity works in practice. Writers with a smaller active vocabulary tend to choose common words and simple grammatical structures, and those choices are exactly what a language model finds predictable. The researchers showed that asking a model to enhance the word choices of the TOEFL essays cut the average false positive rate from 61.22 percent to 11.77 percent. Just 1 of the 91 essays remained unanimously flagged by all seven tools. The same trick also helps genuine AI text slip past detectors, which reveals that these tools partly reward fancy vocabulary instead of detecting authorship. Our primer on bias and fairness metrics shows how to measure that kind of disparity in any classifier.

An ethical detection practice builds in safeguards before the first accusation, not after the first mistake. Disclose to writers which tools are used and what happens with their scores. Never let a score serve as the sole evidence for a penalty, and give the writer a real chance to respond with drafts, notes, and an explanation. Track outcomes by group, because if second-language writers are flagged at a much higher rate in your institution, that finding is itself evidence that the process is biased. Also consider the position of neurodivergent writers and anyone who relies on grammar assistance, who may produce unusually regular prose for legitimate reasons.

Watermarks and Provenance as a Technical Alternative

Moving on from statistical guesswork, watermarking changes the problem by building a detectable signal into the text at the moment of generation. The Kirchenbauer scheme hashes the previous token to split the vocabulary into a green list and a red list. It then nudges the model toward green tokens at every step. A detector later counts how many tokens landed on the green list and computes a z-score, which converts the count into a p-value. Human text should land on the green list about as often as chance predicts, so a very high z-score is hard to explain without the watermark. The demonstration script in the how-to section below produces a z-score near 0.86 for plain synthetic text and near 19.19 for biased text.

Google DeepMind’s SynthID-Text shows both the promise and the limits of this approach in production. The team ran a live experiment on 20 million Gemini prompts, where half of the responses carried the watermark. User thumbs-up and thumbs-down feedback showed similar satisfaction with both versions. The limits are just as clear in the same report. Heavy editing or asking another chatbot to summarize the text would likely obscure the signal. Only Google and authorized developers can access the detector, and open models can be modified to remove watermarks. Watermarks therefore answer a narrow question, which is whether this exact text came from this specific cooperating model, and they say nothing about text from other sources. Meta has taken a similar route for video, as covered in our report on its watermarking tool for AI videos.

Detection in Classrooms, Newsrooms, and Hiring

Among the settings where detection is used, classrooms attract the most attention and the most controversy. Vanderbilt concluded in 2023 that AI detection software is not an effective tool and should not be used, citing opacity, privacy concerns, and bias alongside the false positive math. Many instructors now prefer to redesign assessment, using in-class writing samples, staged drafts, oral defenses, and assignments tied to personal experience or local data. Those methods produce process evidence that no detector can supply. The broader institutional picture is covered in our analysis of universities navigating academic disruption from AI.

Newsrooms and publishers face a different version of the problem because their risk is reputational. An editor wants to know whether a freelance pitch or contributed column was machine drafted, and whether any facts were fabricated along the way. The practical safeguard is verification, such as confirming every quotation with its source and checking that cited studies exist, rather than relying on a classifier. Wikipedia’s volunteers took the same path, building a human field guide to AI writing habits while declaring automated tools unreliable. For editors writing their own rules, see how one major outlet approached ethical use of AI.

In hiring, an automatic rejection based on a detector score is the riskiest use of all, because it can discriminate against non-native applicants without anyone noticing. Cover letters and writing samples are exactly the short, formal, template-influenced documents on which detectors perform worst. A better approach is to treat the written application as one signal. Verify skills through a live exercise, a work trial, or a structured interview where the candidate explains their reasoning. If a team does use a detector, it should log the false positive rate for the relevant text length and have a person review every flag. Employers should also take legal advice, because disparate impact on protected groups can create liability even when no one intended it.

Privacy, Policy, and Legal Exposure

Given the sensitivity of student essays and job applications, pasting text into a free detector is itself a data transfer that deserves scrutiny. Vanderbilt raised exactly this concern, noting that sending student work to third-party tools involves unknown privacy and data use policies. Depending on location and context, regulations such as FERPA in the United States or GDPR in Europe may govern what an institution can share and retain. Ask each vendor whether submitted text is stored, whether it is used to train models, and how deletion requests are handled. Our guide to safeguarding student privacy with AI tools lists practical questions to include in vendor reviews.

A written policy that defines permitted AI use and guarantees due process is the best protection against both misconduct and wrongful accusation. The policy should say what counts as acceptable assistance, such as brainstorming or grammar checking, and what must be disclosed. It should state that detector output is advisory, name who reviews cases, specify what evidence can be considered, and describe how an accused person can appeal. Institutions that skip these steps expose themselves to complaints and to reputational damage when a false accusation becomes public. The wider ethical context appears in our discussion of ethical issues of AI in education.

Common Mistakes That Wreck Detection Accuracy

On top of the technical limits, avoidable human errors account for many bad calls. The first mistake is testing text that is too short, since Turnitin itself raised its minimum from 150 to 300 words to reduce errors. The second is pasting content that includes quotations, references, code, or boilerplate, which distorts statistical measures and can produce high or low scores that say nothing about the author. The third is ignoring the base rate, so that a flag in a class where almost no one cheats is trusted far more than the arithmetic supports. A fourth is running a single tool once and treating the result as final, when repeated runs on edited versions can differ.

Asking a chatbot whether it wrote a given passage is one of the least reliable methods, because chatbots have no memory of text they produced elsewhere. The model will usually respond with a confident-sounding opinion based on style, which is exactly the weak signal this guide warns about. Another common trap is fixating on single telltale words in the text. The author of TechCrunch’s summary noted that claims that words like “delve” or “underscore” reveal AI are poorly supported, and they become less reliable as models improve. Better evidence comes from clusters of habits combined with verifiable facts and process records.

Finally, many reviewers confuse a detector’s confidence with the author’s guilt. A tool can be highly confident and still be wrong, particularly on the unusual human voices it has not encountered. Reviewers also forget that students often use AI in ways the policy permits, which makes the question of authorship less binary than a percentage suggests. Our coverage of how students use ChatGPT to cheat and the new cheating dilemma shows how varied the behavior is. Match your response to the actual behavior, and let the evidence, not the percentage, set the tone of the conversation.

The Future of AI Text Detection and Authorship

Looking ahead, the arms race between generators and detectors has a theoretical ceiling that explains why many researchers are pessimistic about pure detection. Sadasivan and colleagues connect the best possible detector’s AUROC to the total variation distance between the distribution of human text and machine text. As models improve, that distance shrinks, and the room for any detector to separate the two shrinks with it. This does not mean detection is impossible today, since models still have quirks, but it means the quirks are a wasting asset. Detectors that depend on them will need constant retraining and will stay vulnerable to attacks.

The more durable direction is provenance, which records how a text was made instead of guessing afterward. That includes watermarks from cooperating model providers, signed content credentials in the style of the C2PA standard, and writing environments that keep a verifiable revision history. Each of these has costs, including privacy concerns about keystroke logging, the need for industry-wide adoption, and the reality that open models can be run without any watermark. Process-based assessment in education, where students show drafts and reasoning, is the human-scale version of the same idea. I expect the practical future to be a mix of weak statistical signals, strong provenance where available, and well-designed assignments.

Trust in written work will increasingly depend on transparent norms about disclosure rather than on any tool that can prove a negative. The broader consequences are already visible in the erosion of confidence online, which we examined in AI undermining online trust. Scientific publishing has felt the pressure too, as shown by reports of AI-generated science flooding academic journals. Readers, editors, and teachers who learn the skills in this guide will be better placed than those who outsource judgment to a dashboard. The goal is not to catch every machine-written sentence but to make well-calibrated decisions when the stakes justify the effort.

How to Detect AI Generated Text Accurately in Six Steps

Building on the workflow described above, the six steps below turn the theory into commands you can run on any document. Each step produces a number or a note that goes into your evidence log, and none of them produces a verdict by itself. The scripts use only the Python standard library, apart from the perplexity step, which needs a small open model. Pro tip: run every step on a known human sample from the same author or course, so you can see what normal looks like before judging the disputed text. Treat the outputs as prompts for questions, never as proof, and keep the log so that a colleague can reproduce your reasoning.

Step 1 - Gather the text and set the ground rules

Start by obtaining the original file instead of a pasted excerpt, and save a copy before touching it. Check the word count, because passages under about 300 words give unstable scores and Turnitin raised its own minimum from 150 to 300 words for that reason. Remove quotations, reference lists, code, and boilerplate that the author did not write in their own voice. Write down the policy that applies, including which kinds of AI assistance were allowed for this assignment or task. Finally, collect two or three earlier samples of the same person's writing, since a comparison with their own baseline is far stronger than a comparison with a generic model.

wc -w essay_017.txt
wc -w earlier_sample_1.txt earlier_sample_2.txt

Step 2 - Measure the linguistic signals with a short script

Next, run the signal script below to measure sentence rhythm, vocabulary variety, repeated phrasing, and stock expressions. It needs no installation, and it prints a dictionary of 8 values you can paste into your evidence log. Burstiness here is the coefficient of variation of sentence length, meaning the standard deviation divided by the mean, so higher values indicate a more varied rhythm. The type-token ratio compares unique words to total words, and the repeated trigram share shows how much of the text reuses the same three-word sequences. The stock phrase list is deliberately short, because a long list of banned words produces false alarms.

import re
import statistics
import sys
from collections import Counter

STOCK_PHRASES = [
    "it is important to note", "plays a crucial role", "a testament to",
    "in the ever-evolving", "a pivotal moment", "emphasizing the significance",
    "reflecting the continued relevance", "rich tapestry", "delve into",
    "in summary", "navigating the complexities",
]

def split_sentences(text):
    parts = re.split(r"(?<=[.!?])\s+", text.strip())
    return [p for p in parts if len(p.split()) > 2]

def signals(text):
    sentences = split_sentences(text)
    lengths = [len(s.split()) for s in sentences]
    words = re.findall(r"[A-Za-z']+", text.lower())
    paragraphs = [p for p in re.split(r"\n\s*\n", text) if p.strip()]
    para_lengths = [len(p.split()) for p in paragraphs]
    trigrams = Counter(zip(words, words[1:], words[2:]))
    repeated = sum(c for c in trigrams.values() if c > 1)
    lowered = text.lower()
    hits = [p for p in STOCK_PHRASES if p in lowered]
    mean_len = statistics.mean(lengths)
    return {
        "sentences": len(sentences),
        "mean_sentence_words": round(mean_len, 1),
        "burstiness_cv": round(statistics.pstdev(lengths) / mean_len, 2),
        "type_token_ratio": round(len(set(words)) / len(words), 2),
        "repeated_trigram_share": round(repeated / max(1, len(words) - 2), 3),
        "paragraph_length_cv": round(
            statistics.pstdev(para_lengths) / statistics.mean(para_lengths), 2
        ) if len(para_lengths) > 1 else None,
        "stock_phrase_hits": hits,
        "word_count": len(words),
    }

if __name__ == "__main__":
    print(signals(open(sys.argv[1], encoding="utf-8").read()))

The two sample files used in this guide show how the output looks, and they are far shorter than the 300 words you need for a real judgment. The first is a casual human anecdote and the second is a generic chatbot-style paragraph about time management. On these tiny samples, the human text scored a burstiness of 0.58 with no stock phrases, while the machine-style text scored 0.17 and matched four stock phrases. Real documents will not separate so cleanly, so compare against the same author's earlier samples rather than against these numbers. Do not use any fixed cutoff from this script as a verdict, because the ranges overlap heavily on real writing.

python3 signals.py human.txt
python3 signals.py ai.txt

{'sentences': 10, 'mean_sentence_words': 15.3, 'burstiness_cv': 0.58, 'type_token_ratio': 0.77, 'repeated_trigram_share': 0.0, 'paragraph_length_cv': 0.2, 'stock_phrase_hits': [], 'word_count': 157}
{'sentences': 9, 'mean_sentence_words': 12.8, 'burstiness_cv': 0.17, 'type_token_ratio': 0.74, 'repeated_trigram_share': 0.0, 'paragraph_length_cv': 0.05, 'stock_phrase_hits': ['it is important to note', 'plays a crucial role', 'a testament to', 'in summary'], 'word_count': 116}

Step 3 - Score perplexity with a small reference model

Moving to the statistical layer, perplexity needs a reference language model, and the open GPT-2 model is a convenient choice because it runs on an ordinary laptop. The script below follows the sliding-window method described in the Hugging Face perplexity guide, so long documents are scored in overlapping windows. It converts the text into tokens, a process explained in our primer on tokenization in LLMs, and then averages the negative log probability the model assigns to each one. You need the torch and transformers packages installed, and the first run downloads the model weights. I did not execute this particular script in the environment used to write this guide, so test it on a known sample before relying on it.

import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL).eval()

def perplexity(text, stride=512):
    ids = tokenizer(text, return_tensors="pt").input_ids
    window = model.config.n_positions
    nll_sum, token_count, prev_end = 0.0, 0, 0
    for begin in range(0, ids.size(1), stride):
        end = min(begin + window, ids.size(1))
        target_len = end - prev_end
        chunk = ids[:, begin:end]
        labels = chunk.clone()
        labels[:, :-target_len] = -100
        with torch.no_grad():
            loss = model(chunk, labels=labels).loss
        nll_sum += loss.item() * target_len
        token_count += target_len
        prev_end = end
        if end == ids.size(1):
            break
    return math.exp(nll_sum / token_count)

if __name__ == "__main__":
    import sys
    print(round(perplexity(open(sys.argv[1], encoding="utf-8").read()), 1))

Interpretation is where most people go wrong with this number. Lower perplexity means the text is more predictable to GPT-2, which is weak evidence of machine authorship. A perplexity of 40 under one model cannot be compared with 40 under another. Always score the disputed text and the author's earlier samples with the same model, then look at the gap rather than the absolute value. Newer chatbots are trained differently from GPT-2, so a more recent reference model may separate the classes better. Our explainer on how language models are trained shows why training data shapes these differences. Formal, edited, or non-native writing will all score low, which is why this signal only ever adds to the log and never closes a case.

python3 gpt2_ppl.py essay_017.txt
python3 gpt2_ppl.py earlier_sample_1.txt

Step 4 - Run two independent detectors and log them

Now bring in dedicated detectors, choosing two that belong to different families so their errors are not simply copies of each other. A supervised commercial classifier plus a zero-shot method such as Binoculars is a sensible pairing, and many institutions already have access to one of them. For each tool, record the 5 essentials: the version, the date, the threshold applied, the raw score, and the vendor's documented false positive rate at that threshold. Do not average the scores, because they are on different scales and mean different things. If the two tools disagree sharply, treat that as a sign that the text is short, edited, translated, or from an unknown model. In that case rely more heavily on the other steps.

import datetime
import json

entry = {
    "date": datetime.date.today().isoformat(),
    "document": "essay_017.docx",
    "word_count": 612,
    "signals": {"burstiness_cv": 0.21, "stock_phrase_hits": 3, "perplexity_gpt2": 18.4},
    "detectors": [
        {"name": "Detector A", "family": "supervised", "threshold": 0.5,
         "score": 0.72, "documented_fpr": 0.01},
        {"name": "Detector B", "family": "zero-shot", "threshold": 0.9,
         "score": 0.64, "documented_fpr": 0.0001},
    ],
    "process_evidence": [],
    "level": "moderate",
}

with open("evidence_log.jsonl", "a", encoding="utf-8") as log:
    log.write(json.dumps(entry) + "\n")

The values in the sample entry are illustrative and not from a real case. What matters is the structure, because a log like this lets a second reviewer see exactly what was run and under which assumptions. Write the log before you speak to the author, since it is easy to remember evidence selectively once a conversation has started. Store the logs securely, because they may contain student or applicant information that falls under privacy rules. A line-per-case format also makes it easy to review error patterns across a term or hiring round, which supports the fairness checks described earlier.

Step 5 - Work out what a flag is worth and check for watermarks

Before acting on any flag, compute how likely it is to be correct given the prevalence of AI use in your setting. The short script below applies Bayes rule to three scenarios, using a detector with 80 percent sensitivity. At 2 percent prevalence and a 1 percent false positive rate, a flag is right about 62 percent of the time. At 10 percent prevalence the same tool is right about 90 percent of the time. When a 5 percent false positive rate meets 5 percent prevalence, a flag is correct only about 46 percent of the time, which is worse than a coin flip. The interactive calculator near the top of this article lets you try your own numbers.

def precision(prevalence, sensitivity, false_positive_rate):
    true_flags = prevalence * sensitivity
    false_flags = (1 - prevalence) * false_positive_rate
    return true_flags / (true_flags + false_flags)

for prevalence, fpr in [(0.02, 0.01), (0.10, 0.01), (0.05, 0.05)]:
    p = precision(prevalence, 0.80, fpr)
    print(f"prevalence={prevalence:.0%} fpr={fpr:.0%} -> flag is correct {p:.0%} of the time")

prevalence=2% fpr=1% -> flag is correct 62% of the time
prevalence=10% fpr=1% -> flag is correct 90% of the time
prevalence=5% fpr=5% -> flag is correct 46% of the time

If the text may come from a model that applies a statistical watermark, you can test for one when you hold the key or have access to an official detector. The demonstration below implements the green-list z-test from the Kirchenbauer scheme on synthetic tokens, which shows the principle without needing a real model. On my run, plain synthetic text produced a z-score of 0.86 and biased text produced 19.19, and any score above roughly 4 is very unlikely by chance. A real deployment such as SynthID uses a different construction and a restricted detector, so this snippet teaches the idea and is not a tool for checking Gemini output. Absence of a watermark proves nothing, because most text carries none.

import hashlib
import math
import random

GAMMA = 0.25  # fraction of the vocabulary placed on the green list

def is_green(prev_token, token, key="demo-key"):
    digest = hashlib.sha256(f"{key}|{prev_token}|{token}".encode()).digest()
    return int.from_bytes(digest[:8], "big") / 2**64 < GAMMA

def z_score(tokens, key="demo-key"):
    pairs = list(zip(tokens, tokens[1:]))
    greens = sum(is_green(a, b, key) for a, b in pairs)
    n = len(pairs)
    return (greens - GAMMA * n) / math.sqrt(n * GAMMA * (1 - GAMMA))

def fake_generate(vocab, length, bias, key="demo-key", seed=1):
    rng = random.Random(seed)
    out = [rng.choice(vocab)]
    while len(out) < length:
        candidates = rng.sample(vocab, 8)
        green = [c for c in candidates if is_green(out[-1], c, key)]
        out.append(rng.choice(green) if green and rng.random() < bias else rng.choice(candidates))
    return out

vocab = [f"w{i}" for i in range(2000)]
plain = fake_generate(vocab, 200, bias=0.0)
marked = fake_generate(vocab, 200, bias=0.9)
print("unwatermarked z =", round(z_score(plain), 2))
print("watermarked   z =", round(z_score(marked), 2))

unwatermarked z = 0.86
watermarked   z = 19.19

Step 6 - Verify facts, gather process evidence, and write the finding

Finally, move from numbers to the human checks that carry the most weight. Pick 3 factual claims and 2 citations in the text, then confirm each against the original source, since fabricated references are among the strongest indicators of unchecked machine output. Ask for process evidence such as the document's revision history, earlier outlines, notes, and the sources the author consulted. Invite the author to explain a few choices in the text in a short conversation, which usually reveals ownership quickly. Combine everything using a fixed rule so that two reviewers reach the same level from the same evidence. Then write the result as a likelihood statement instead of a judgment of guilt. The function below encodes the four levels used in this guide, and you should adapt the thresholds to your own policy.

def concern_level(linguistic_signals, detector_flags, process_evidence):
    if process_evidence:
        return "strong"
    if linguistic_signals >= 1 and detector_flags >= 1 and linguistic_signals + detector_flags >= 3:
        return "moderate"
    if linguistic_signals + detector_flags >= 1:
        return "weak"
    return "none"

Key Insights

• OpenAI's own classifier identified only 26 percent of AI-written text and falsely flagged 9 percent of human writing, which led the company to withdraw it on July 20, 2023.
• A Stanford-led study found seven detectors flagged an average of 61.22 percent of 91 TOEFL essays as AI-generated, so second-language writers carry most of the false positive burden.
• At the scale of 75,000 papers, even Turnitin's claimed 1 percent false positive rate implies roughly 750 students could be wrongly flagged in a single year.
• Binoculars reports catching over 90 percent of machine text at a 0.01 percent false positive rate without any training data, though that result comes from curated benchmark datasets.
• The RAID benchmark scored 12 detectors on more than six million generations and 11 adversarial attacks and found them easily fooled by sampling changes, repetition penalties, and unseen models.
• Google tested SynthID-Text on 20 million live Gemini prompts and saw similar user satisfaction with watermarked answers, which suggests that watermarking costs little in perceived quality.
• Turnitin analyzed 38.5 million submissions in six weeks and found 3.5 percent contained more than 80 percent AI writing, which shows how widely these tools now touch student work.

Taken together, these numbers describe a field where laboratory performance and real-world reliability have drifted far apart. Detectors can look impressive on curated data, as Binoculars and DetectGPT do. They lose accuracy when text is edited, paraphrased, translated, or written by someone with a smaller English vocabulary. Base rate arithmetic turns small error rates into large absolute numbers of wrongly flagged people, which is exactly what Vanderbilt calculated. Watermarking offers a cleaner statistical signal but only for cooperating models and only until the text is rewritten. The sensible response is a layered workflow that treats every score as a probability and builds conclusions from several independent kinds of evidence.

Dimension	Supervised classifier	Zero-shot statistical	Watermark detection	Stylometry and human review
Needs training data	Yes, large labeled sets of human and machine text	No, uses pre-trained reference models	No, uses a secret key and a hashing rule	No, relies on reader skill and baselines
Works on any model's output	Only families seen in training	Broadly, with some drop on unseen models	Only text from the cooperating model	Yes, any source
Transparency of the verdict	Low, a score with little explanation	Medium, a probability-based statistic	High, a z-score and p-value	High, specific habits can be quoted
Robustness to paraphrasing	Weak	Weak to moderate	Weak once text is heavily rewritten	Moderate, factual checks survive rewriting
False positive risk for non-native writers	High, as shown on TOEFL essays	High, since simple prose is predictable	Low, human text rarely carries the key	Moderate, depends on reviewer bias
Computing cost	Low once deployed	Medium to high, needs one or two LLMs	Low at detection time	High in reviewer time
Best use	Quick triage of large volumes	Second opinion from a different family	Provenance for a cooperating model	Resolving contested cases
Main weakness	Degrades as new models appear	Fooled by edits and memorized text	Needs adoption and detector access	Slow, subjective, hard to scale

Detection Methods in Practice: Three Worked Examples

Turning to concrete practice, three worked examples show how the methods described above behave when someone actually runs them. Each example uses a different family of technique, from a statistical watermark test to commercial detector comparisons. The point is not to rank products, since tool performance changes with every release. The useful lesson is that identical methods give very different answers depending on the text, the generator, and the threshold. Read each example for its limitation as carefully as for its headline number.

A Green-List Watermark Test on Synthetic Tokens

I implemented a toy generator over a synthetic vocabulary of 2,000 words and ran a green-list test modeled on the Kirchenbauer watermark scheme. Each token's greenness came from a SHA-256 hash of the previous token and the candidate, with a quarter of the vocabulary counted as green. The unbiased generator produced 200 tokens and a z-score of 0.86, which is the unremarkable value that chance predicts. The biased generator preferred green candidates 90 percent of the time and produced a z-score of 19.19 over the same length. That gap shows why a watermark supports a far stronger statistical statement than any perplexity threshold, since a z-score above roughly 4 is very unlikely by chance. The limitation is that this is a synthetic toy with a known key, so it says nothing about real chatbots or the proprietary design of SynthID. Rewriting the tokens would still dilute the green fraction and push the score back toward zero.

Decrypt's Declaration of Independence Test

Decrypt ran the preamble of the Declaration of Independence through four commercial detectors to see whether a famous human text would pass. ZeroGPT labeled it 97.93 percent AI-generated, while QuillBot returned human-written 100 percent and GPTZero gave an 89 percent probability of human authorship. The reporters then tested an excerpt of the 1909 story "The Machine Stops" alongside a ChatGPT-4o rewrite of it. GPTZero separated the two versions cleanly, rating the original 97 percent human and the rewrite 95 percent AI. ZeroGPT scored the original at 4.27 percent AI and the rewrite at only 6.35 percent, so it effectively treated the machine version as human. The limitation is that this was a single journalistic test with a handful of texts, so it illustrates variability and does not measure true error rates.

AIMultiple's Eighteen-Text Detector Benchmark

AIMultiple built a small benchmark by running ten popular detectors over nine human-written passages and nine passages generated by ChatGPT, Gemini, and DeepSeek. A text counted as flagged when a tool scored it at 50 percent or higher for AI likelihood. Across all tools, 88 percent of the human texts and 71 percent of the AI texts were classified correctly. Accuracy varied by generator, with 87 percent for ChatGPT texts, 72 percent for DeepSeek, and just 54 percent for Gemini. Individual tools differed enormously, from Sapling's 45 percent false positive rate to GPTZero flagging none of the human texts at all. The limitation is sample size, because 18 texts of 100 to 196 words carry no confidence intervals, and one misclassification moves a tool's score by more than 10 percentage points. Treat the results as a demonstration of variance rather than a leaderboard.

Lessons From Real Detection Failures and Successes

Stepping back from isolated tests, three longer case studies show how a vendor, a university, and a model maker handled the same limits over time. All three stories end with some version of the same conclusion, which is that detector output needs context before it can support a decision. Institutions that publish their reasoning, as these three did, give everyone else a chance to learn from the mistakes. Each case below is described through its problem, its response, its measurable result, and the criticism that remains. Read them as a sequence, because each one built on the lessons of the one before.

Case Study: OpenAI's Withdrawn AI Text Classifier

In January 2023 educators were struggling with a flood of ChatGPT essays, and they needed some way to tell machine writing from student work. OpenAI faced pressure to help because its own product had created the problem. The company launched an AI text classifier designed to estimate the likelihood that a passage came from an AI system rather than a person. The design mirrored the supervised approach described earlier in this guide, with a threshold deciding when a passage received a flag. Teachers and administrators watched closely, because a reliable tool from the maker of ChatGPT would have settled a great deal of debate.

Performance proved disappointing, because the classifier correctly identified only 26 percent of AI-written text while wrongly labeling 9 percent of human text as machine written. A tool that misses nearly three quarters of AI text and wrongly flags roughly one in eleven humans cannot support disciplinary decisions. OpenAI withdrew the tool on July 20, 2023, citing its low rate of accuracy. Commentators asked why anyone should believe other detectors work if the company behind ChatGPT could not build a reliable one. A fair limitation on that reading is that this was an early public tool and commercial products use different training, so the numbers do not transfer directly. Still, the episode set the tone for years of skepticism about detection claims.

Case Study: Vanderbilt Switches Off Turnitin's Detector

When Turnitin switched on AI writing detection in April 2023, Vanderbilt faced a problem. It received less than 24 hours' notice, and customers could not initially turn the feature off. The university's teaching center evaluated the tool over the following months. Turnitin claimed a 1 percent false positive rate, and Vanderbilt applied that number to the 75,000 papers it had submitted in 2022. The arithmetic implied that roughly 750 student papers could be wrongly flagged as partly AI written in a single year. Staff also noted that Turnitin had not explained in detail how the tool decided, saying only that it looked for patterns it did not define. On August 16, 2023 the university adopted a new policy, disabling the detector for the foreseeable future and publishing guidance for instructors.

The decision removed a source of potential wrongful accusations affecting hundreds of students and shifted attention to assignment design and open conversation about acceptable AI use. The guidance also raised privacy concerns about sending student work to third-party tools whose data policies were unknown. It cited research showing that detectors more often mislabel writing by non-native English speakers. The decision was contested, since Turnitin argued its document-level false positive rate was below 1 percent for texts with substantial AI content, and many institutions kept using the tool. A limitation of the analysis is that it applied a headline rate to every paper, whereas the real figure depends on text length and AI share. Even so, the case remains the clearest public example of an institution turning base rate math into policy, as the Vanderbilt guidance lays out in full.

Case Study: Turnitin Recalibrates After 38.5 Million Submissions

Turnitin launched its AI writing indicator in April 2023 and faced a problem almost immediately, because real classrooms behaved differently from its laboratory tests. Within six weeks the system had analyzed about 38.5 million submissions, and 3.5 percent contained more than 80 percent AI-written text. Educators reported false positives, especially when only a small share of a document was flagged. The company's own analysis found more false positives when the detected AI share fell below 20 percent, and it called this inconsistent behavior that it was still investigating. It also found that the opening and closing sentences of documents were scoring unreliably.

The solution was a set of calibration changes rolled out over the following weeks. Turnitin added an asterisk to scores under 20 percent, raised the minimum length from 150 to 300 words, and changed how it scores the first and last sentences. After the changes it reported a document-level false positive rate below 1 percent for documents with more than 20 percent AI writing. Its own product update puts the sentence-level rate at about 4 percent. Those are meaningful improvements, yet the company acknowledged that it cannot fully mitigate the risk of false positives. A 4 percent sentence-level rate remains a real limitation when a single highlighted sentence is used as evidence. The case is useful because it shows a vendor measuring, admitting, and adjusting in public.

Frequently Asked Questions About How to Detect AI Generated Text Accurately