Browsing: bots

OpenAI, Google, Microsoft and Anthropic have agreed to support a new way for website owners to stop their content being used to train AI models. The companies are backing a shared opt-out signal in a website’s robots.txt file, which tells AI crawlers whether they’re allowed to use its content for training.