# Cloudflare Crawl Endpoint Hacker Approved page: https://hackerapproved.com/p/cloudflare-crawl-endpoint/ 7 people recommended Cloudflare Crawl Endpoint in Hacker News comments, 3 had mixed feelings and 11 criticized it, November 2025 to October 2026. It was recommended in 2 of 12 months, in 2 threads. It does not meet the Hacker Approved badge rule. 6 of the 7 people who recommend it commented in one thread, “Cloudflare Crawl Endpoint”. **Verdict:** More people criticize it than recommend it. Lets developers pull structured data from public pages without managing session state, though critics say it only works on sites that already allow bots and doubt its robots.txt handling. Summary written by Claude from the comments below. ## What people warn about - Only works on sites allowing bots (3 comments in the sample): "in practice it only works on sites that already allow bots to crawl them." ([source](https://news.ycombinator.com/item?id=47350062)) "they are NOT proxies.. that means majority of the popular sites actually block scraping" ([source](https://news.ycombinator.com/item?id=47331092)) "Doesn't work for pages protected by cloudflare in my experience." ([source](https://news.ycombinator.com/item?id=47333106)) - Relies on Cloudflare's trust promises (3 comments in the sample): "Cloudflare promise not to do that, so if we take their word for it, it's a quite different setup." ([source](https://news.ycombinator.com/item?id=47330962)) "Is this just a way to strong-arm non-cloudflarians into adopting their platform if you don't want your site crawled?" ([source](https://news.ycombinator.com/item?id=47336533)) "they are selling scraping countermeasures but are now selling scraping too" ([source](https://news.ycombinator.com/item?id=47330134)) - Robots.txt compliance doubts (2 comments in the sample): "their crawler will respect the robots.txt dissalowed user-agents, right after the section hat explains how to change your user-agent" ([source](https://news.ycombinator.com/item?id=47332728)) "yet another opt out bot which you need your web server to match on special behaviour to block" ([source](https://news.ycombinator.com/item?id=47330766)) ## Why people recommend it - Enthusiastic reception from commenters (3 comments in the sample): "This plus a /crawl endpoint is genius." ([source](https://news.ycombinator.com/item?id=47335261)) "Cloudflare getting all the cool toys" ([source](https://news.ycombinator.com/item?id=47330197)) "This might be really great!" ([source](https://news.ycombinator.com/item?id=47330118)) - Simplifies structured data pulling from pages (1 comment in the sample): "Something that helped me was Cloudflare's new crawler API." ([source](https://news.ycombinator.com/item?id=49020285)) ## Where to get it - [Visit the official site](https://developers.cloudflare.com/browser-run/) ## The badge A product earns the Hacker Approved badge when at least 10 people recommend it, recommenders outnumber critics at least 2 to 1 (mixed feelings count toward neither side), it was recommended in at least 4 of the 12 months and in at least 5 different threads, and no single thread supplies more than half of the people who recommend it. Health and money products are never badged. ## Cite as Hacker Approved, "Cloudflare Crawl Endpoint", https://hackerapproved.com/p/cloudflare-crawl-endpoint/ (7 recommend, 3 mixed, 11 criticize; Hacker News comments, November 2025 to October 2026). Counts updated 2 Nov 2026, with data through October 31, 2026.