Well, After spending a year selling bot-blocking tools, Cloudflare just dropped a /crawl endpoint that lets you ingest an entire website with a single API call
Haha! What a irony!
Imagine the company that spent all of last year protecting the internet from AI scrapers just handed developers the keys to scrape it all.
On March 10, 2026, Cloudflare launched the new /crawl endpoint as part of its Browser Rendering service.
For anyone who builds data-heavy applications or takes on freelance web scraping projects, this is a massive paradigm shift.
You no longer need to manage brittle Puppeteer scripts, bypass CAPTCHAs, or pay exorbitant per-page fees to third-party APIs.
You just send one curl command to Cloudflare, and they return the entire site.
Here is a breakdown of why the developer community is calling this a massive “U-Turn,” how the API actually works, and why it is about to replace so many entire local scraping stacks
The Great Cloudflare U-Turn
To understand the controversy, you have to look at what Cloudflare did last year.
So, basically, In mid-2025, they launched “AI Crawl Control” and “Pay Per Crawl,” allowing publishers to block AI bots or charge them per request.
They published wild stats showing how Anthropic crawled sites 73,000 times for every one referral link clicked.
They positioned themselves as the shield.
Now, they are selling the sword.
And the communities are being crazy.
The Hacker News and Reddit communities are rightfully pointing out the mob-like irony: Cloudflare is selling the scraping countermeasures to the publishers, and now selling the scraping infrastructure to the developers.
Because Cloudflare controls the DNS and proxy layer for a massive chunk of the internet, their edge network is uniquely positioned to crawl without getting blocked.
The Tech: One API Call, 100,000 Pages
So, how does it actually work? It is brutally simple.
You send a POST request with a starting URL, and Cloudflare spins up a headless Chrome instance on their edge network.
- Automatic Discovery: The crawler automatically finds URLs via sitemaps and page links.
- Depth & Scale: You can set the depth and limit up to 100,000 pages.
- Output Formats: It executes the JavaScript (perfect for heavy Vue.js or React SPAs) and returns the output in HTML, Markdown, or structured JSON.
- Workers AI Integration: If you request JSON, you can pass a schema, and Cloudflare’s Workers AI will automatically extract the exact data points you need from the rendered DOM.
If you are scraping static sites, there is even a render: false mode that skips the headless browser entirely for lightning-fast, cheap extraction.
Killing the Scraping SaaS Market
For a developer taking on freelance data extraction gigs, the infrastructure overhead is usually the hardest part.
Setting up Playwright on a local M3 Max is fine for a small test, but scaling it requires managing proxies and rotating IP addresses.
SaaS platforms like Firecrawl and Crawl4AI stepped in to solve this, but their pricing models are steep.
Firecrawl charges around $47/month for 100k pages.
Cloudflare just nuked that pricing model. Because they bill based on browser-hours ($0.09/hour) rather than per-page, high-volume crawls are exponentially cheaper.
Even on the Workers Free plan, you get 10 minutes of browser time a day, which is enough to run 5 crawl jobs of up to 100 pages each.
So, Normally, this requires writing a custom Python script with BeautifulSoup, dealing with pagination, and hoping the target site doesn’t aggressively block my AWS IP.
With the new /crawl endpoint, you just fire this off in your terminal:
curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
-H 'Authorization: Bearer <apiToken>' \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example-sports-venues.com",
"formats": ["json"],
"jsonOptions": {
"prompt": "Extract venue name, hourly price, and court type"
}
}'No proxy management. No DOM parsing. Just pure data extraction.
Cloudflare’s decision to launch a mass-crawling tool might be ethically blurry given their anti-bot posturing last year, but from an engineering perspective, it is a masterpiece.
If you are building RAG pipelines, migrating CMS content, or hunting for freelance scraping contracts, this is one among the most powerful tool dropped this year.
Let the big companies fight over the ethics of “Pay Per Crawl.”
For developers in the trenches, data just got a whole lot cheaper to acquire.