Developer tools · distributed computing framework
Apache Hadoop
Recommended as a known quantity that really works for large-scale work; critics say it is overkill for smaller scale, with a large upfront cost.
Summarized from 25 comments.
Recommended as a known quantity that really works for large-scale work; critics say it is overkill for smaller scale, with a large upfront cost.
- recommend
- 8
- criticize
- 17
- months recommended
- 3 of 13
Why people love it
Suited to large-scale workloads · 3
Spark and Hadoop have their place, maybe not in rinky dink startup land
Jan 2026 · commentSpark and Hadoop have their place
Jan 2026 · commentHadoop ought be used for hot data
Jan 2026 · commentReliable, well-known in practice · 2
MR and Hadoop etc. would take care of recovering from operational failures like disk or network outages
Nov 2025 · commentHadoop is a known quantity and really works.
Jan 2026 · comment
What people warn about
Overkill for small-scale needs · 3
you're definitely paying a large upfront cost for scalability you very much might not need
Jan 2026 · commentI did not have the heart or courage to tell them during the demo that this was overkill
Sep 2026 · commentthat has trade-offs that typically aren't worth it if you don't need that scale
Jan 2026 · commentDated, hard-to-use MapReduce design · 3
Back when Hadoop came along you built operators and these operators communicated through disk
Apr 2026 · commentHadoop is still a Map/Reduce clone, thus destined to fail at useability.
Jan 2026 · commentthe Hadoop era Java stuff
Jan 2026 · commentFrustrating, poorly supported in practice · 3
you will get Hadoop levels of non-application support needed
Jul 2026 · commentthat was just an exercise in frustration
Mar 2026 · commentIts the worst of all possible worlds.
Jan 2026 · comment
What hackers pick instead
- Apache SparkFree open-source engine for big data processing that people say just works. Critics say it is overkill for small datasets and hard to run yourself.18recommend
- Ray distributed computing frameworkPeople recommend Ray for scaling Python workloads across machines, especially ML infrastructure, with built-in autoscaling. Critics say clusters add heavy overhead and cost, and that it is slow.10recommend
Every quote
The 18 most detailed of 25 comments. Each links to the original.
MR and Hadoop etc. would take care of recovering from operational failures like disk or network outages
Recommends · Nov 2025 · in “I want a good parallel language [video]”
Spark and Hadoop have their place, maybe not in rinky dink startup land
Recommends · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
Hadoop is a known quantity and really works.
Recommends · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
Spark and Hadoop have their place
Recommends · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
Hadoop ought be used for hot data
Recommends · Jan 2026 · in “Internet Archive's Storage”
Back when Hadoop came along you built operators and these operators communicated through disk
Criticizes · Apr 2026 · in “The Economics of Software Teams: Why Most Engineering Orgs Are Flying Blind”
you're definitely paying a large upfront cost for scalability you very much might not need
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
I did not have the heart or courage to tell them during the demo that this was overkill
Criticizes · Sep 2026 · in “118M Queries per Second on Neki”
that has trade-offs that typically aren't worth it if you don't need that scale
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
Hadoop is still a Map/Reduce clone, thus destined to fail at useability.
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
you will get Hadoop levels of non-application support needed
Criticizes · Jul 2026 · in “Terrence Tao's ChatGPT Conversation about the Jacobian Conjecture Counterexample”
that was just an exercise in frustration
Criticizes · Mar 2026 · in “I'm 60 years old. Claude Code has ignited a passion again”
Its the worst of all possible worlds.
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
the Hadoop era Java stuff
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
the Hadoop stuff is particularly annoying given the relatively poor quality (IMO) of the Hadoop code base
Criticizes · Mar 2026 · in “Hardwood: A New Parser for Apache Parquet”
So Hadoop was doing distributed compute wrong but now they have it figured out?
Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”
a single linux box could get done 10x faster with some basic pipes
Criticizes · Apr 2026 · in “r/programming bans all discussion of LLM programming”
for some reason hadoop I guess
Criticizes · Nov 2025 · in “What the hell have you built”
Counts from 1,795,640 Hacker News comments, Nov 2025 to Nov 2026. How we count · Report a mistake