Are AI detectors accurate? What the published tests show

The short answer

AI text detectors can be right on clean, unedited machine text in the conditions they were built for. They are much weaker on the text people actually argue about: second-language writing, edited machine drafts and short pieces. None of the tests above supports using a score on its own to decide who wrote something.

The question has two halves: how often a detector catches machine text, and how often it accuses a human. For a writer or a student, the second half is the one that matters.

How this guide reads the studies: method

This guide counts four published studies, dated 2023 to 2024, that each tested more than one AI text detector against text whose origin the researchers knew in advance. Every figure is taken as the paper states it, from its abstract or results section, and the study and year are named next to it.

What this guide did not do: mashshare ran no tests of its own, submitted no text to any detector, and does not rank or recommend any product. It leaves out vendor accuracy claims, which are not independent tests, and image detectors. Detectors are updated often, so a 2023 result describes a tool as it was in 2023; the direction of the findings has held, but the exact numbers belong to their test date.

What each study found

Weber-Wulff and colleagues, 2023: fourteen tools

The broadest classroom-style test, by a team led by Debora Weber-Wulff, was published in 2023 in the International Journal for Educational Integrity and is available as Testing of Detection Tools for AI-Generated Text. The researchers ran 54 test cases through 12 publicly available tools and 2 commercial systems, using human-written text, chatbot output, machine-translated text and machine text edited by hand or by a paraphrasing tool.

Weber-Wulff and colleagues (2023) found that even the highest accuracy among the 14 tools was below 80 percent. The authors concluded that the tools are neither accurate nor reliable. They also found a main bias towards calling text human-written, so machine text was missed more often than it was caught, and that obfuscation such as paraphrasing made performance significantly worse.

Liang and colleagues, 2023: non-native writers

A Stanford team led by Weixin Liang tested seven widely used detectors on 91 essays written by non-native English speakers for the TOEFL exam, and on essays by US eighth-grade students. The study was published in Patterns in 2023, and the preprint of the Liang study on detector bias against non-native English writers is open to read.

Liang and colleagues (2023) found an average false positive rate of 61.22 percent on the TOEFL essays. In the same 2023 study, 89 of the 91 human-written essays were flagged as AI-generated by at least one detector, while the native essays were classified almost perfectly. The authors link the bias to more constrained vocabulary and phrasing, which makes text more predictable to a language model. One result is hard to forget: when the researchers had a chatbot enrich the vocabulary of the same human essays, the average false positive rate fell from 61.22 to 11.77 percent.

Perkins and colleagues, 2024: edited machine text

A team led by Mike Perkins at British University Vietnam tested detectors on machine text altered to evade them, plus human-written controls, in GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education, published in 2024 in the International Journal of Educational Technology in Higher Education.

Perkins and colleagues (2024) report a mean accuracy of 39.5 percent on unedited machine-generated text across 805 tests. The same 2024 study found only 67 percent of tests on the human-written controls were accurate. After adversarial edits such as adding spelling errors or varying sentence length, average accuracy fell to 22.14 percent. The authors conclude that the tools cannot currently be recommended for deciding whether an academic integrity violation has occurred.

Dugan and colleagues, 2024: the RAID benchmark

The largest test is a benchmark rather than a single experiment. RAID, a shared benchmark for machine-generated text detectors, led by Liam Dugan at the University of Pennsylvania and released in 2024, holds over 6 million generations from 11 models across 8 domains, with 11 adversarial attacks and 4 decoding strategies.

Dugan and colleagues (2024) evaluated 8 open-source and 4 closed-source detectors and found them easily fooled by adversarial attacks, changes in sampling strategy, repetition penalties and models they had not seen. Their key finding concerns reporting: detectors reached the high accuracies quoted in viral reports only at similarly high false positive rates. RAID therefore compares detectors at a fixed false positive rate of 5 percent, and the authors argue that fixing and disclosing that rate is crucial.

Why accuracy figures disagree

Three things move an accuracy figure more than the detector itself. The first is the threshold. A detector outputs a score, and someone decides where flagging starts; set it low and the tool catches more machine text but accuses more humans, which is the trade-off the RAID authors (2024) put at the centre of their benchmark. The second is the test set. Clean, unedited output from a known model is easy; translated, edited or short text is not, as Weber-Wulff and colleagues (2023) and Perkins and colleagues (2024) both showed. The third is the writer. Liang and colleagues (2023) showed that the same detectors that handled native essays well failed on non-native ones.

A fourth factor sits outside the tool: how common machine text is in the pile being checked. When most texts are human, even a small false positive rate produces many wrongful flags, as the example below shows.

A worked example: one term, 200 essays

Suppose a department runs a detector over 200 essays in one term. Every number in this example is invented for illustration. Say 20 essays were in fact machine-written and 180 were not, and the detector catches 14 of the 20 while wrongly flagging 9 of the 180. That is a 5 percent false positive rate, the level at which the RAID benchmark (2024) compared detectors.

The detector raises 23 flags in the example, and 9 of them point at students who wrote their own work. A flagged student therefore has about a two in five chance of being innocent, even though the tool is right about 95 percent of human essays. If only 5 of the 200 essays were machine-written, the same tool would catch 3 or 4 of them and still wrongly flag about 10 honest writers, so most flags would be false. The tool has not changed; only the mix has.

What a detector score can and cannot support

• It can support: a reason to look more closely, ask the writer about the work and check drafts, sources and history.
• It cannot support: a finding on its own, a penalty, a public accusation, or a claim about which tool was used.
• It says least about: short text, edited text, translated text and writing by non-native speakers. For social posts, the guide to whether an AI detector can judge a social post covers why length matters.
• A clean score proves little too: Weber-Wulff and colleagues (2023) and Perkins and colleagues (2024) both found that light edits let machine text through.

If you are the person flagged, the evidence that settles authorship is your process, not another score: revision history, dated drafts, notes and sources. The guide to being accused of posting AI content sets out what to gather and how to answer. Publishers can make disputes rarer by putting a named author on every piece, using byline schema, and by stating tool use plainly in a disclosure line that survives a share.

Pitfalls

• Quoting a vendor's accuracy figure without its false positive rate. The RAID authors (2024) found headline accuracies appear only when false positives are high.
• Treating formal or simple English as a signal. Liang and colleagues (2023) found exactly that writing flagged most often.
• Assuming a 2023 result describes a tool today, or that an update fixed the problem. Check whether any newer independent test exists before relying on either view.
• Reading a low score as proof of human authorship. Edited machine text often passes, as Perkins and colleagues (2024) showed.

The provenance section covers the signals that hold up better than a score: bylines, disclosure lines, revision records and, for images, Content Credentials.

Sources

• Weber-Wulff et al., Testing of Detection Tools for AI-Generated Text (2023)
• Liang et al. (2023), detectors of machine-written text are biased against non-native English writers
• Perkins et al. (2024), GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Education
• Dugan et al. (2024), RAID: a shared benchmark for machine-generated text detectors

Questions

Related reading

1. 01Can an AI detector judge a social post?Why AI text detectors are least reliable on the short, informal text of social posts and captions, what published tests found, and what to rely on instead.5 min read
2. 02Accused of posting AI content: how to show your workWhat to do when readers, a client or a platform say your post or article was written by AI: the evidence that settles it and what does not help.5 min read
3. 03AI disclosure lines that survive a shareWhere to put an AI assistance disclosure so it reaches readers who only see the share card: titles, descriptions, bylines and the first lines of a post.5 min read