Situation Report
The threat is volume. OpenAI announced a new classifier on Monday at a press briefing in Washington, DC. It’s built to help child safety investigators find real abuse cases buried under a growing pile of AI-generated images, according to The Verge AI. The tool detects child sexual abuse material (CSAM) and also flags whether an image was likely made by AI.
That second flag is the important part. Investigators don’t only need to know an image is abusive. They need to know whether a real child might be in danger right now.
The Numbers Behind the Problem
The National Center for Missing and Exploited Children (NCMEC) runs the CyberTipline, the main US system for collecting and sorting CSAM reports. As The Verge AI reports, it received 400,000 reports tied to generative AI in 2025. In 2024 it got 67,000. That’s roughly a sixfold jump in one year.
Researchers have warned this could make NCMEC’s workload “unmanageable.” North Carolina Attorney General Jeff Jackson co-chairs the AG Alliance’s AI Task Force, and he spoke at OpenAI’s event. He called the influx “explosive.” He also warned that if it keeps growing this way, “it is going to mean that we fail in our mission.”
What OpenAI Is Deploying
- A new CSAM classifier. OpenAI says it flags 48 percent fewer images than earlier approaches. The images it does flag are twice as likely to contain abusive material. In plain terms, fewer false alarms and better hits.
- An AI-generated content flag. OpenAI’s head of child safety public policy, John Buckley, said it’s the first broadly available CSAM classifier with a specific flag for AI-generated material. It’s meant to catch output from many sources, including open-source models and nudifying apps.
- Free access for vetted organizations. Qualified groups can use the tool at no cost.
- Developer guidance. OpenAI is publishing more detailed advice on how to detect this type of content.
- Law enforcement training. OpenAI will help train investigators on how AI can support their cases.
Changes Inside ChatGPT
ChatGPT’s image generator already refuses CSAM requests. Now OpenAI is adding consequences:
- Users who try get a new warning, written with expert input, that points them toward help.
- A first offense triggers a 24-hour lockout from ChatGPT.
- The warning says the account could be closed permanently.
- Support resources are offered to help stop the behavior.
OpenAI says it reports “all instances of CSAM” to NCMEC. Buckley said the 24-hour pause follows expert advice and “international evidence on what works.” The goal, he said, is “to interrupt harmful behavior and direct people toward specialist help to stop it.”
Assessment
What stands out here is the shift from blocking to triage. For years, detection tools answered one question: is this abusive? AI-generated imagery adds a second one: is the child real? “That triage is critical in law enforcement reporting flows,” Buckley said.
Buckley was also clear that fake doesn’t mean harmless. AI-generated CSAM “potentially erodes social norms in offending against children,” he said, and it’s often found alongside imagery of real children. The classifier helps investigators prioritize. It doesn’t make synthetic material acceptable.
The timing isn’t an accident either. OpenAI is under real pressure:
- Last month eight states sued OpenAI over alleged harms to children, including claims that ChatGPT encouraged illegal drug use and suicide.
- The Take It Down Act now requires platforms to remove real or AI-generated nonconsensual intimate images within 48 hours.
- The newly elected Congress may push harder on AI companies, especially after Grok’s undressing spree earlier this year.
- The US government treats both AI-generated and real CSAM as illegal, and 48 states have laws covering sexually explicit AI deepfakes.
A free, well-built safety tool lets OpenAI show it’s being proactive while regulators are deciding how hard to push.
What to Watch
- Adoption. Free access only matters if trust and safety teams and NCMEC partners actually plug it in.
- Independent validation. The 48 percent and 2x figures come from OpenAI. Outside testing will show whether they hold up.
- Industry pressure. A free benchmark tool raises the bar for other model makers, especially open-source projects and image apps with weaker safeguards.
- The AG partnership model. Jackson said most AG work happens “after the fact” and called this joint prevention effort “still basically an experiment.” If it works, expect more state-level partnerships with AI labs.
If you build with image generation, now’s a good time to review your own detection and reporting pipeline. Full details are in the original report from The Verge AI.