Digiday·2026-08-14
European publishers are getting hit harder by AI bot scraping, report finds
European publishers are getting hit harder than North American sites by AI bot scraping – and no one really knows why. A recent report from TollBit found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules – which websites use to communicate to crawlers what they can and cannot scrape – ignored more often. Here are the main findings from that r
一份报告指出,欧洲出版商遭受人工智能机器人抓取的影响比北美网站更为严重,但其具体原因尚不明确。TollBit近期发布的一份报告发现,欧洲网站更容易被AI机器人抓取,获得的推荐流量更少,并且其robots.txt规则(网站用于向爬虫程序传达哪些内容可抓取、哪些不可抓取的协议)被忽视的频率更高。以下是该报告的主要发现: