Cloudflare Crawl Endpoint
Web crawler API in Hosting & cloud
Lets developers pull structured data from public pages without managing session state, though critics say it only works on sites that already allow bots and doubt its robots.txt handling.
Lets developers pull structured data from public pages without managing session state, though critics say it only works on sites that already allow bots and doubt its robots.txt handling.
22 people gave an opinion
- recommend it
- 11
- criticize it
- 11
- approval share
- 50%
Recommended in 2 of 13 months
- November 2025: not recommended
- December 2025: not recommended
- January 2026: not recommended
- February 2026: not recommended
- March 2026: recommended
- April 2026: not recommended
- May 2026: not recommended
- June 2026: not recommended
- July 2026: recommended
- August 2026: not recommended
- September 2026: not recommended
- October 2026: not recommended
- November 2026: not recommended (month in progress)
Why it doesn’t carry the badge
- 11 people recommend itNeeds 10 or more, met.
- 1 recommender for every criticNeeds 2 to 1 or better, not met.
- Recommended in 2 of 13 monthsNeeds 4 or more, not met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Simplifies structured data pulling from pages
“for pulling structured data from public pages this is a big simplification over managing session state yourself”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “Something that helped me was Cloudflare's new crawler API.”
Comment from 23 Jul 2026 in “Back to Kagi” Enthusiastic reception from commenters
“This plus a /crawl endpoint is genius.”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “Cloudflare getting all the cool toys”
Comment from 10 Mar 2026 in “Cloudflare Crawl Endpoint” “This might be really great!”
Comment from 10 Mar 2026 in “Cloudflare Crawl Endpoint”
What people warn about
Each mark is one comment
Only works on sites allowing bots
“in practice it only works on sites that already allow bots to crawl them.”
Comment from 12 Mar 2026 in “Cloudflare Crawl Endpoint” “they are NOT proxies.. that means majority of the popular sites actually block scraping”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “Doesn't work for pages protected by cloudflare in my experience.”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” Relies on Cloudflare's trust promises
“Cloudflare promise not to do that, so if we take their word for it, it's a quite different setup.”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “Is this just a way to strong-arm non-cloudflarians into adopting their platform if you don't want your site crawled?”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “we are the gate keeper so that the Mafia boss decide what will be the intermediate that is proper for itself”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” - Comment from 10 Mar 2026 in “Cloudflare Crawl Endpoint”
Robots.txt compliance doubts
“their crawler will respect the robots.txt dissalowed user-agents, right after the section hat explains how to change your user-agent”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint” “yet another opt out bot which you need your web server to match on special behaviour to block”
Comment from 11 Mar 2026 in “Cloudflare Crawl Endpoint”
What people said
20 of the 22 comments that mention it, newest first. Each links to the original comment.
“Something that helped me was Cloudflare's new crawler API.”
“in practice it only works on sites that already allow bots to crawl them.”
“their own /crawl endpoint lists "training AI systems" as a use case alongside regular crawling”
“Is this just a way to strong-arm non-cloudflarians into adopting their platform if you don't want your site crawled?”
“This plus a /crawl endpoint is genius.”
“this gets your content out without hammering production”
“Doesn't work for pages protected by cloudflare in my experience.”
“their crawler will respect the robots.txt dissalowed user-agents, right after the section hat explains how to change your user-agent”
“I don't see any mention of WARC as an output format.”
“they are NOT proxies.. that means majority of the popular sites actually block scraping”
“Cloudflare promise not to do that, so if we take their word for it, it's a quite different setup.”
“Sounds pretty useless for any serious AI company”
“yet another opt out bot which you need your web server to match on special behaviour to block”
“for pulling structured data from public pages this is a big simplification over managing session state yourself”
“we are the gate keeper so that the Mafia boss decide what will be the intermediate that is proper for itself”
“Cloudflare is just skating to where the puck is going to be on this one.”
“it'd basically mean that the only people who can't crawl are those without money”
“Cloudflare getting all the cool toys”
“they are selling scraping countermeasures but are now selling scraping too”
“This might be really great!”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.