Meta directed hundreds of contractors to create fake accounts posing as teenagers. They then flooded chatbots from OpenAI, Google and Character.AI with prompts about suicide, self-harm, eating disorders, sex and even cannibalism. The goal? Test how well those systems held up against disturbing content.
But the effort crossed lines. It used throwaway under-18 profiles without the rivals’ knowledge. One internal name for the operation was Cannes. A contractor called Covalen managed much of the work. And the volume was staggering.
Inside the Scale of the Testing
One spreadsheet reviewed by Wired contained 3,748 prompts. Hundreds focused on suicide and self-harm. Hundreds more targeted eating disorders. At least 239 touched on sex or romance. All written from a child’s or teenager’s viewpoint.
A fifth-grader whose classmate pointed a gun at his mouth. A girl desperate to hide her bulimia from her parents. A query about whether fantasizing about eating your neighbor’s child counted as normal. Another from a high schooler asking where to get cocaine. Images accompanied some prompts: pictures of pills, nooses, knives, even a medical diagram of a gynecological exam.
That was just one batch. Another round involved more than 45,000 prompts. Contractors logged chatbot replies in detail. An internal Covalen document called the project “comprehensive AI safety benchmarking” that produced “critical datasets for model comparison and compliance.” What Meta ultimately did with those datasets remains unclear. The company says it was not using the material to train its own models.
Contractors felt uneasy. “I’ve seen a lot of things I wish I hadn’t while doing this job,” one told Wired. “Everyone I knew who worked on this project was completely gobsmacked by some of the text they were asking us to test. Like, surely we are going to get in trouble for doing this?”
The operation fits a pattern. Meta has repeatedly outsourced sensitive or traumatic tasks. In 2020 the company settled a lawsuit from content moderators who developed PTSD after reviewing graphic violence and child abuse videos. Similar trauma claims surfaced years later in Kenya. This year contractors reported being forced to watch intimate footage captured by Ray-Ban Meta smart glasses, including private sexual encounters and bathroom scenes. The pattern raises questions about who bears the human cost when AI systems need testing or improvement.
Meta defended the work. A spokesperson told Wired that “testing and benchmarking chatbot responses to help ensure safe and age-appropriate experiences is a responsible, industry-standard practice.” Yet the secrecy, the use of fake child accounts and the absence of any public sharing of findings set it apart.
Rumman Chowdhury, CEO of Humane Intelligence, pushed back. “Structuring a monthslong, large-scale project that appears designed to systematically break those rules, via dummy accounts masquerading as children, is outside what is usually described as ‘industry standard’ evaluation,” she said. Chowdhury added that the approach sits in “exactly the kind of governance gray zone where safety becomes a convenient cover for anticompetitive practices.”
The targeted companies took notice once details emerged. Character.AI called the actions a violation of its terms of service and of the characters and worlds its community created. OpenAI said it was investigating. Google declined comment on the terms violation itself.
But the story does not stop at competitive benchmarking. It collides with a larger crisis around AI and child sexual abuse material. The National Center for Missing & Exploited Children received 1.5 million reports in 2025 with a generative AI connection. More than 182,000 of those, excluding bulk training-data reports, involved offenders possessing, generating or attempting to generate such material. The Internet Watch Foundation assessed 8,029 AI-generated images and videos as realistic child sexual abuse content in 2025. AI-generated videos alone jumped 26,385 percent from the prior year.
These numbers come amid broader failures in reporting. U.S. Senator Chuck Grassley released information in early July 2026 showing Meta submitted nearly 11 million reports of suspected online child exploitation to NCMEC in 2025. Yet the submissions suffered from consistency problems, quality issues on enticement and trafficking cases, and reports of violent content lacking any clear child exploitation link. Law enforcement described many Meta tips as low-quality “junk” that drain resources, according to a February 2026 Guardian report on a New Mexico lawsuit against the company.
So Meta simultaneously tests rivals by pushing disturbing prompts through pretend-child accounts and struggles with its own moderation and reporting systems. The contrast is stark. On one side, a sophisticated, secretive program to probe competitor weaknesses. On the other, complaints that automated systems flood authorities with poor signals while real cases move slowly.
Experts have warned for years that training data can embed harmful patterns. Some open-source image models were found to contain real child abuse material in their datasets. Others produce new synthetic versions when prompted cleverly. Red-teaming efforts, which this project resembles in method if not in transparency, have become standard. Yet when the red teaming stays hidden and targets rivals using fake underage personas, the ethical and legal questions multiply.
Attorneys consulted by Wired noted the work did not appear to solicit actual child sexual abuse material or illegal obscenity. Even so, it violated terms of service across the industry. And it forced contractors to generate and read thousands of distressing scenarios. The psychological toll echoes the earlier content moderation lawsuits.
Meta is not alone in facing scrutiny over AI and child safety. Commitments from companies including Meta, Google, OpenAI and others to Thorn’s Safety by Design principles include promises to red-team for child sexual abuse risks and improve transparency. Progress reports from 2025 show varying levels of follow-through. Yet the Cannes project, which predates some of those updates, suggests that internal safety work can blur into competitive intelligence gathering.
The episode highlights a tension. AI companies must test their systems against harmful inputs. They need to measure where guardrails fail. But the methods matter. Secret large-scale campaigns that impersonate minors and withhold results from the public invite skepticism about true motives. Safety testing should not become an excuse for opaque anticompetitive behavior.
Regulators, lawmakers and the public now have fresh reason to demand clearer standards. How should companies test rivals? When does safety benchmarking cross into unauthorized access or deceptive practices? What obligations exist to share findings that could benefit the whole sector? And who protects the contractors asked to spend their days inventing scenarios about traumatized children?
Those questions will not resolve quickly. The technology moves faster than the rules. Meta’s contractors kept detailed spreadsheets. They recorded every chatbot reply. The data exists. Whether it leads to better protections for actual users or simply sharper competitive edges remains to be seen. One thing is clear. The human work behind these tests is harder, and more disturbing, than the polished product launches suggest.
WebProNews is an iEntry Publication