Common Crawl
Hacker ApprovedWeb crawl dataset
Free web archive that people use instead of crawling sites themselves; critics say it is sampled, monthly and less fresh than rivals.
Free web archive that people use instead of crawling sites themselves; critics say it is sampled, monthly and less fresh than rivals.
26 people gave an opinion
- recommend it
- 19
- criticize it
- 7
- approval share
- 73%
Recommended in 9 of 13 months
- November 2025: recommended
- December 2025: not recommended
- January 2026: recommended
- February 2026: recommended
- March 2026: recommended
- April 2026: recommended
- May 2026: recommended
- June 2026: not recommended
- July 2026: recommended
- August 2026: recommended
- September 2026: recommended
- October 2026: not recommended
- November 2026: not recommended (month in progress)
Why it carries the badge
- 19 people recommend itNeeds 10 or more, met.
- 2.7 recommenders for every criticNeeds 2 to 1 or better, met.
- Recommended in 9 of 13 monthsNeeds 4 or more, met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Saves crawling the web yourself
“I feel using something that already exists will be more cooperative and considerate than everyone overloading every website with their spider.”
Comment from 24 Feb 2026 in “Crawling a billion web pages in just over 24 hours, in 2025” “I think we can get by with using CommonCrawl, pulling every few months the fresh content and updating the search stubs.”
Comment from 8 Jan 2026 in “Why we're taking legal action against SerpApi's unlawful scraping” “they'd just pull down a copy of Common Crawl like every other major data analysis project”
Comment from 3 May 2026 in “Videolan Dav2d” Free, open data for research
“public exchange of data is great for innovation and research”
Comment from 8 Sep 2026 in “Navier-Stokes – Tristan Buckmaster [pdf]” “great resource. Common crawl is a goldmine”
Comment from 16 Apr 2026 in “IPv6 Traffic Crosses the 50% Mark for the First Time” Good quality pages, little spam
“the data that I’ve experimented with from 2014 seemed high quality, with very little spam”
Comment from 15 Apr 2026 in “Ask HN: What funding models exist for a search engine?” “I was surprised by how many of them seemed high quality”
Comment from 21 May 2026 in “Show HN: My independent search engine focused on user control”
What people warn about
Each mark is one comment
Sampled, monthly, less fresh than rivals
“probably doesn't make sense on its own due to low update frequency (monthly) and somewhat limited crawl scope”
Comment from 4 May 2026 in “Google is discontinuing its free web search index for developers” “Common Crawl is a sample of the web, so it's not that directly helpful”
Comment from 26 Apr 2026 in “Ask HN: Scaling a targeted web crawler beyond 500M pages/day” “CC is not on the same scale as Google and not nearly as fresh.”
Comment from 8 Jan 2026 in “Why we're taking legal action against SerpApi's unlawful scraping” - Comment from 21 Sep 2026 in “Spain orders blocks on Archive.today and its mirrors”
Site owners object to scraping
“I'm not interested in participating in your project, block CCBot in robots.txt, and have requested deletion of my data”
Comment from 15 Feb 2026 in “Internet Increasingly Becoming Unarchivable” “They outsource their dirty and ethically questionable data collection to a "non-profit,"”
Comment from 7 Nov 2025 in “The Company Quietly Funneling Paywalled Articles to AI Developers” “its product is unlawful reproductions of copyrighted data for commercial use”
Comment from 9 Jun 2026 in “US publishers tell Common Crawl to stop scraping and delete archive” - Comment from 15 Feb 2026 in “Internet Increasingly Becoming Unarchivable”
What people said
28 of the 30 comments that mention it, newest first. Each links to the original comment.
“is reasonably reliable but not as comprehensive as archive.org”
“public exchange of data is great for innovation and research”
“FYI for those needing a list of domains”
“good luck redacting Common Crawl”
“Common Crawl is a really cool project”
“It's easy to opt out or limit what is collected.”
“We agree that it would be great if it was even more widely used.”
“the solution is a better common crawl”
“its product is unlawful reproductions of copyrighted data for commercial use”
“Each should have the pieces needed to construct this kind of view from the data.”
“CommonCrawl seems like a good idea”
“I was surprised by how many of them seemed high quality”
“probably doesn't make sense on its own due to low update frequency (monthly) and somewhat limited crawl scope”
“they'd just pull down a copy of Common Crawl like every other major data analysis project”
“Common Crawl is a sample of the web, so it's not that directly helpful”
“great resource. Common crawl is a goldmine”
“the data that I’ve experimented with from 2014 seemed high quality, with very little spam”
“You could probably extract a lot from”
“they are really lacking quality data sources if you know where to look”
“Common Crawl has free egress”
“I feel using something that already exists will be more cooperative and considerate than everyone overloading every website with their spider.”
“Received April 2024. I have not been advised. Please advise.”
“I'm not interested in participating in your project, block CCBot in robots.txt, and have requested deletion of my data”
“CC is not on the same scale as Google and not nearly as fresh.”
“I think we can get by with using CommonCrawl, pulling every few months the fresh content and updating the search stubs.”
“it includes archives of pages that don’t exist anymore”
“They outsource their dirty and ethically questionable data collection to a "non-profit,"”
“Not enough people know about common crawl, or know how to write multithreaded scrapers”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.