Skip to content

Tools

ChatGPT adds 24-hour suspensions for seeking child sexual abuse material

Edited and reviewed by AI Ping Editorial Team

3-line summary

  1. OpenAI is rolling out warnings that end ChatGPT conversations it identifies as attempts to find child sexual abuse material. Some interventions include a 24-hour account suspension.
  2. The feature is rolling out worldwide on the web and mobile for personal accounts. Users can appeal a temporary suspension through their notification email.
  3. OpenAI also announced a separate image classifier, available at no cost to vetted organizations, to help human reviewers prioritize flagged material.
PracticalOffice workersDevelopers
A dark smartphone in front of a frosted glass divider
AI-generated conceptual image: A phone and a frosted glass divider represent a ChatGPT safeguard that ends a harmful conversation. · AI Ping

OpenAI announced on November 16 that ChatGPT will end conversations it identifies as attempts to find child sexual abuse material (CSAM). An intervention may also suspend account access for 24 hours. OpenAI’s help center says the warnings and suspensions are rolling out worldwide for personal accounts in Chat and Work modes on the web and mobile.

OpenAI also announced a separate image classifier for organizations that work on child safety. The warning changes what happens in a ChatGPT conversation. The classifier helps organizations put images that may require attention in front of a reviewer sooner.

OpenAI already prohibits child sexual abuse material and trains its models to reject requests for it. The update adds a product warning and, in some cases, a break in access. The warning describes the harm of sexualizing children, including in AI-generated material, and directs people toward specialist support to stop harmful behavior.

A warning ends the chat and a suspension triggers an email

When the warning appears, the conversation ends. For a signed-in user, the conversation is hidden from the history and cannot be reopened. OpenAI’s help center says a warning can also come with a temporary account suspension. The announcement specifies a 24-hour pause. OpenAI says the appearance of the warning can vary by country.

If the account is suspended, the user is signed out and an email explains the reason, the length of the suspension and any available next steps. The email contains a link to contest the temporary suspension. Filing an appeal does not automatically restore access. Once the period ends, the user can sign in again if no other account restriction is active.

A temporary suspension and a permanent closure are separate decisions. OpenAI says repeated violations can lead to longer suspensions or account closure. Its help center says the latter can follow a separate review that finds a policy violation. A user who receives a closure notice alongside a notice of temporary suspension can request a review of the decision identified in each email.

A blue-sand hourglass beside a closed laptop
AI-generated conceptual image: An hourglass and a closed laptop represent a temporary pause in ChatGPT account access. · AI Ping

An image score helps reviewers decide where to start

OpenAI is making the classifier available at no cost through its API to organizations it vets. Platforms can identify known material by checking incoming files against previously confirmed images. The new classifier adds a way to flag potentially abusive images that have not been identified before, including ones made with generative AI.

The classifier scores how likely an image is to be abusive material, with a separate score for potentially AI-generated abusive material. A platform with many incoming images can use those signals to decide which items in its review queue need attention first. OpenAI says the National Center for Missing & Exploited Children helped it develop and evaluate the tool and prepare it for deployment.

OpenAI’s model card says a score alone cannot establish that an image should be reported. It calls for review by a trained, authorized person before making a report based on a flag from the classifier. Known-image matching and existing reporting procedures remain part of the organization’s response.

A reddish card beside a tray containing plain gray cards
AI-generated conceptual image: The card set aside from the tray represents an image prioritized for human review. · AI Ping

What the evaluation measured

In a comparison using images uploaded to ChatGPT, OpenAI estimated that the classifier would flag 48% fewer images overall while identifying roughly as many abusive images as its existing tool. The model card says an evaluation dataset can differ from what a platform actually receives. The figure is an estimate from the ChatGPT comparison, and an organization with different traffic may see different results.

OpenAI also released child-safety guidance for developers. It covers ways to let users report concerns, processes for assessing reports and support for the people who review them. It is aimed at organizations building a reporting process as well as those with one in place.

OpenAI said it had begun participating in Lantern, a program run by the Tech Coalition. Through Lantern, participating companies exchange signals about possible online child exploitation. The coalition says a company that receives a signal investigates it under its own policies and determines what action to take.

Sources