Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out of being scraped
Culture Index
Score Breakdown
Relevance
9/25
Freshness
20/25
Authority
18/20
Brand Signal
16/15
Depth
4/15
5-Axis Cultural Radar
Right now, if you don’t want your site to be scraped by AI companies, you have to block each company’s crawlers individually. On Friday, Anthropic, OpenAI, Google, and Microsoft said they would support a common robots.txt signal that, per OpenAI, “makes it easier for publishers and website operators to communicate the same preference to multiple...
