Cloudflare also introduced a new “pay per crawl” model, which allows publishers to charge AI crawlers for content access. A close-up of a server array powering a cloud-services system.

Cloudflare’s new “pay per crawl” initiative has sparked a debate among SEO professionals and digital marketers. The company has introduced a default AI crawler-blocking system alongside new ...

Dozens more are also now blocking the crawler of Common Crawl, a major source of AI training data. Unique, high quality data, mainly scraped from the web, is vital to the performance of AI models.