OpenAI to support a shared robots.txt signal for AI training preferences
2026-11-01 · OpenAI
OpenAI to support a shared robots.txt signal for AI training preferences
OpenAI has announced that it will honor a common robots.txt signal that lets publishers and website operators express preferences about the use of their content for foundational generative AI training.
What changes
- A shared signal will let website operators communicate the same preference to multiple participating providers without maintaining separate rules for each one.
- OpenAI aims to implement the signal in the coming months and will publish implementation details separately.
- The new signal will not replace OpenAI's existing robots.txt crawler controls, which will continue to apply and be respected.
Standards background
The effort draws on the IETF AI Preferences working group. OpenAI says it will continue participating in standards discussions and encourages other AI providers to honor the signal.
The announcement does not yet provide the exact syntax or full implementation details. Publishers should wait for the technical update while retaining their existing crawler controls.