OpenAI adds AI-training flag to robots.txt
What is Making it easier for websites to express AI training preferences?
Websites can now embed a special directive in their robots.txt to tell OpenAI and other participating providers whether their pages may be used for model training. OpenAI says it will honor that signal when crawling the site.
Who it is for
Site owners who want to control if their content ends up in future AI models should add the new directive.