The sample length and where each detector cuts off the input need to be reported. That was the first thing that confused me about these comparisons. A detector may behave very differently on a 150-word answer than on a 1,500-word essay, and some sites may only analyze part of a long document. If every tool is not judging the same amount of text, the percentages are not directly comparable.
Raw ChatGPT output still seems useful as a basic check, but I would not use it to rank the tools. It is basically the easiest possible case. It can tell you whether a detector is completely missing obvious generated text, but a perfect result there does not tell you much about normal use. Real submissions may contain rewritten paragraphs, quotations, grammar corrections, personal examples, and sections written by different people.
I agree with @codeninja4096 that human controls are required. What would make the test clearer for me, though, is using paired versions of the same assignment. Start with a human essay on a topic, generate an AI essay from the same prompt, and then create edited and mixed versions from that AI essay. This reduces the chance that one category happens to contain easier topics or a noticeably different writing style. Otherwise the detector might really be recognizing formal essay language, certain subjects, or repetitive prompt wording rather than AI authorship.
I would want the benchmark to include at least these versions for each prompt:
- fully human-written
- raw AI-generated
- AI text with basic grammar and wording edits
- heavily rewritten AI text
- human writing polished by an AI tool
- a mixed document containing both human and AI sections
That fifth category seems easy to overlook. Someone can write the whole draft themselves and use AI only to improve clarity. A detector that marks that as fully generated could create more trouble than a detector that misses some heavily disguised AI text. False accusations matter, especially in education, so the result should show how often genuine human writing gets flagged and how often human writing with minor AI assistance gets overstated.
The test should be blinded, but it should be time-stamped too. These online detectors can change without making the change obvious to users. If the same benchmark is run months later, the scores may move even though the dataset has not changed. Saving the detector name, plan, date, input length, displayed score, and exact rule used to turn that score into “AI” or “human” would make the comparison much easier to repeat.
So yes, raw output is an unreliable benchmark if it is the main evidence. It is fine as the beginner-level test case. A fair ranking needs paired samples, human controls, multiple levels of editing, consistent text lengths, and false-positive results. Without those pieces, a 99% catch rate could mean “excellent detector,” or it could mean “this tool calls almost everything AI.”
