# Apache Spark

18 Hacker News commenters recommended Apache Spark and 25 criticized it, November 2025 to November 2026. It was recommended in 9 of 13 months.

**Verdict:** Free open-source engine for big data processing that people say just works. Critics say it is overkill for small datasets and hard to run yourself.

## Why people recommend it

- Handles petabyte-scale data workloads (2 comments): "we use Spark and Apache Beam for many of our large workloads, but that's typically involving data at the petabyte scale." ([source](https://news.ycombinator.com/item?id=46674637)) "Spark is a way to pool memory for a large dataset with multiple stages" ([source](https://news.ycombinator.com/item?id=45923510))
- Runs without constant attention (2 comments): "the value of things like Spark is that you can run "things" without having to worry about how they run" ([source](https://news.ycombinator.com/item?id=48684560)) "I think Spark was the best tool out there when data engineering started taking off, and it just works" ([source](https://news.ycombinator.com/item?id=45925034))
- Strong for ETL pipelines (2 comments): "I think Spark was the best tool out there when data engineering started taking off, and it just works" ([source](https://news.ycombinator.com/item?id=45925034)) "Spark in Scala does the ETL part of this well." ([source](https://news.ycombinator.com/item?id=48016022))
- Part of a large Apache ecosystem (2 comments): "they build world class software that backs massive systems like elastic search, Kafka, spark" ([source](https://news.ycombinator.com/item?id=48630419)) "Large Apache ecosystem (Spark, Flink, Pinot) is completely missing" ([source](https://news.ycombinator.com/item?id=48253394))

## What people warn about

- Overkill for small or medium data (5 comments): "huge industry push to convince managers and more junior engineers that spark and distributed tools are the correct way to do data engineering" ([source](https://news.ycombinator.com/item?id=46667696)) "A desire to have some huge cloud based solution with AWS, spark bla bla when a python script with pandas in a cron job was faster." ([source](https://news.ycombinator.com/item?id=47737192)) "teams really struggle with problems caused by using technologies they don't need (normally spark or kubernetes or both)" ([source](https://news.ycombinator.com/item?id=47860388))
- Painful to operate and maintain (2 comments): "We've seen teams pick Spark because it's "simpler" and then spend three quarters building plumbing that Presto would've given them for free." ([source](https://news.ycombinator.com/item?id=46666453)) "Spark is such a headache to run that people would rather pay Databricks to do it for them." ([source](https://news.ycombinator.com/item?id=50292489))
- Slower than alternatives on modest data (2 comments): "Spark/Hadoop is really so damn inefficient that something like DuckDB shows where that overhead is" ([source](https://news.ycombinator.com/item?id=50294278)) "Only once the datasets reach the 100GiB to TiB range is trino/spark faster." ([source](https://news.ycombinator.com/item?id=49450598))
- Slow to adopt Scala 3 (2 comments): "Databricks took six years before upgrading their proprietary Spark runtime to Scala 2.13." ([source](https://news.ycombinator.com/item?id=46832195)) "To this day, Spark only supports Scala 2.13 although Scala 3 has been around for years now." ([source](https://news.ycombinator.com/item?id=46091739))

## Alternatives hackers recommend

- [Ray distributed computing framework](https://hackerapproved.com/p/ray.md): 10 recommend
- [Apache Hadoop](https://hackerapproved.com/p/hadoop.md): 8 recommend

## Cite as

Hacker Approved, "Apache Spark", https://hackerapproved.com/p/apache-spark/ (counts from 1,795,640 Hacker News comments, November 2025 to November 2026).