Death By A Million Scrapes: What Publishers Face With The Rapid Increase In AI Scraping

Last week, I wrote about the citation and attribution issues that plague AI chatbots and haunt publishers.

If the rapid, continued growth of scraping is any indication, publishers have a few bulletpoint items to add to their nuanced concerns.

In its State of the Bots Q4 2024 report, Tollbit found that AI scrapes per site doubled from what they were in Q3, with scrapes per page tripling in that time, including 40% more unauthorized scrapes.

“Blocking AI bots via robots.txt remains an insufficient mechanism to prevent unwanted scraping,” the report’s executive summary says. “Despite AI search companies’ claims, TollBit sees that AI bots on average are driving 95.7% less click through traffic than traditional Google search.”

In analyzing sites that had set up its analytics, Tollbit found more than 2 million scrapes in Q4, with deals and shopping content getting the most scrapes, followed by national news and consumer technology.

 (Source: Tollbit)

Danielle Coffey, News/Media Alliance president and CEO, says the data “confirms what publishers have known for years — generative AI chat bots are not providing anywhere near the amount of traffic as traditional search.”

“By illegally scraping our content, repackaging it, and giving it to consumers without adequately directing them to our sites, AI companies are using our own content to undermine our businesses,” Coffey says. “Without web traffic, news and media organizations lose subscription and advertising revenue, and cannot continue to fund the quality work that both AI companies and consumers rely on.”