People hired to rate ChatGPT’s answers are being pulled off projects for using AI to do the rating. 404 Media reported on Tuesday that it reviewed internal documents and spoke with contractors working on OpenAI-related assignments, two of whom said colleagues had been fired or “offboarded” for it.
“It’s pretty much the one thing that will get you kicked off ASAP,” one contractor told the outlet, adding that “in a group of thousands there are tons that have been caught.” One document indicated that some projects involve more than 10,000 contractors.
No detectors, no Grammarly
The rules apply to the reviewers who audit other contractors, too. “Do not use GPTZero or any other AI detection tool. They are not reliable,” one guideline reads, according to IBTimes, which summarized the report. “Reviewers may not use AI either, including Grammarly and AI translation, to review, write feedback, or write comments.” Auditors are told to judge the overall pattern instead of a single clue and not to tell evaluators what tipped them off. Red flags include repetitive phrasing, heavy use of em dashes and unusually fast turnaround.
Two of the sources worked for Mercor, which supplies reviewers for ChatGPT work. “Our contracts strictly prohibit the use of LLMs to complete projects and we enforce that,” a Mercor spokesperson said. “When we confirm an expert has used AI to complete a task, we immediately remove them from the project.” OpenAI did not give a detailed public response.
A former contractor shared a termination letter citing the “authenticity” of their work. “I’m not a bad person or worker. I just needed a little boost and turned to AI to help me which eventually led to my downfall,” the person said.
Why the labs want human raters
The concern behind the rule is model collapse, the tendency of models trained on AI-generated text to get worse over time. A Nature paper found that indiscriminate use of model-generated content can strip information from the original data. Human raters are supposed to supply the signal that synthetic data can’t, so an AI-written rating defeats the purpose of hiring them.
The rating work runs deep. Earlier in September, 404 Media described Project Lily, in which contractors read real ChatGPT conversations and score the replies, including whether the bot is being too sycophantic. Training is on by default for consumer ChatGPT plans unless users opt out.
Google has wrestled with rater quality as well. In late 2024, TechCrunch reported that contractors evaluating Gemini had been told to stop skipping prompts outside their expertise and to rate what they understood instead.
WebProNews is an iEntry Publication