ChatGPT adds 24-hour suspensions for seeking child sexual abuse material
Edited and reviewed by AI Ping Editorial Team
3-line summary
- OpenAI is rolling out warnings that end ChatGPT conversations it identifies as attempts to find child sexual abuse material. Some interventions include a 24-hour account suspension.
- The feature is rolling out worldwide on the web and mobile for personal accounts. Users can appeal a temporary suspension through their notification email.
- OpenAI also announced a separate image classifier, available at no cost to vetted organizations, to help human reviewers prioritize flagged material.
OpenAI announced on November 16 that ChatGPT will end conversations it identifies as attempts to find child sexual abuse material (CSAM). An intervention may also suspend account access for 24 hours. OpenAI’s help center says the warnings and suspensions are rolling out worldwide for personal accounts in Chat and Work modes on the web and mobile.
OpenAI also announced a separate image classifier for organizations that work on child safety. The warning changes what happens in a ChatGPT conversation. The classifier helps organizations put images that may require attention in front of a reviewer sooner.
OpenAI already prohibits child sexual abuse material and trains its models to reject requests for it. The update adds a product warning and, in some cases, a break in access. The warning describes the harm of sexualizing children, including in AI-generated material, and directs people toward specialist support to stop harmful behavior.
A warning ends the chat and a suspension triggers an email
When the warning appears, the conversation ends. For a signed-in user, the conversation is hidden from the history and cannot be reopened. OpenAI’s help center says a warning can also come with a temporary account suspension. The announcement specifies a 24-hour pause. OpenAI says the appearance of the warning can vary by country.
If the account is suspended, the user is signed out and an email explains the reason, the length of the suspension and any available next steps. The email contains a link to contest the temporary suspension. Filing an appeal does not automatically restore access. Once the period ends, the user can sign in again if no other account restriction is active.
A temporary suspension and a permanent closure are separate decisions. OpenAI says repeated violations can lead to longer suspensions or account closure. Its help center says the latter can follow a separate review that finds a policy violation. A user who receives a closure notice alongside a notice of temporary suspension can request a review of the decision identified in each email.
An image score helps reviewers decide where to start
OpenAI is making the classifier available at no cost through its API to organizations it vets. Platforms can identify known material by checking incoming files against previously confirmed images. The new classifier adds a way to flag potentially abusive images that have not been identified before, including ones made with generative AI.
The classifier scores how likely an image is to be abusive material, with a separate score for potentially AI-generated abusive material. A platform with many incoming images can use those signals to decide which items in its review queue need attention first. OpenAI says the National Center for Missing & Exploited Children helped it develop and evaluate the tool and prepare it for deployment.
OpenAI’s model card says a score alone cannot establish that an image should be reported. It calls for review by a trained, authorized person before making a report based on a flag from the classifier. Known-image matching and existing reporting procedures remain part of the organization’s response.
What the evaluation measured
In a comparison using images uploaded to ChatGPT, OpenAI estimated that the classifier would flag 48% fewer images overall while identifying roughly as many abusive images as its existing tool. The model card says an evaluation dataset can differ from what a platform actually receives. The figure is an estimate from the ChatGPT comparison, and an organization with different traffic may see different results.
OpenAI also released child-safety guidance for developers. It covers ways to let users report concerns, processes for assessing reports and support for the people who review them. It is aimed at organizations building a reporting process as well as those with one in place.
OpenAI said it had begun participating in Lantern, a program run by the Tech Coalition. Through Lantern, participating companies exchange signals about possible online child exploitation. The coalition says a company that receives a signal investigates it under its own policies and determines what action to take.
Sources
- November 16 announcement and classifier evaluation — OpenAI
- ChatGPT warnings, temporary suspensions, and appeals — OpenAI Help Center
- Separate review procedures for account suspensions and closures — OpenAI Help Center
- Image classifier model card and trained review — OpenAI Deployment Safety Hub
- Child-safety guidance for developers — OpenAI Developers
- Lantern signal-sharing program — Tech Coalition
- News RSS with the announcement’s publication timestamp — OpenAI
