Apache Spark

What hackers pick instead

• Ray distributed computing frameworkPeople recommend Ray for scaling Python workloads across machines, especially ML infrastructure, with built-in autoscaling. Critics say clusters add heavy overhead and cost, and that it is slow.10recommend
• Apache HadoopRecommended as a known quantity that really works for large-scale work; critics say it is overkill for smaller scale, with a large upfront cost.8recommend

Every quote

The 40 most detailed of 46 comments. Each links to the original.

1. we use Spark and Apache Beam for many of our large workloads, but that's typically involving data at the petabyte scale.

   Mixed · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”

2. the value of things like Spark is that you can run "things" without having to worry about how they run

   Recommends · Jun 2026 · in “An oral history of Bank Python (2021)”

3. I think Spark was the best tool out there when data engineering started taking off, and it just works

   Mixed · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

4. they build world class software that backs massive systems like elastic search, Kafka, spark

   Recommends · Jun 2026 · in “Deno Desktop”

5. Databricks took six years before upgrading their proprietary Spark runtime to Scala 2.13.

   Mixed · Jan 2026 · in “The Sovereign Tech Fund Invests in Scala”

6. Spark is a way to pool memory for a large dataset with multiple stages

   Recommends · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

7. Large Apache ecosystem (Spark, Flink, Pinot) is completely missing

   Mixed · May 2026 · in “dotNET (OK, C#) gets union types”

8. Spark in Scala does the ETL part of this well.

   Mixed · May 2026 · in “DAG Workflow Engine”

9. huge industry push to convince managers and more junior engineers that spark and distributed tools are the correct way to do data engineering

   Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”

10. We've seen teams pick Spark because it's "simpler" and then spend three quarters building plumbing that Presto would've given them for free.

    Criticizes · Jan 2026 · in “ClickHouse vs. StarRocks vs. Presto vs. Trino vs. Apache Spark”

11. A desire to have some huge cloud based solution with AWS, spark bla bla when a python script with pandas in a cron job was faster.

    Criticizes · Apr 2026 · in “I run multiple $10K MRR companies on a $20/month tech stack”

12. teams really struggle with problems caused by using technologies they don't need (normally spark or kubernetes or both)

    Criticizes · Apr 2026 · in “Laws of Software Engineering”

13. Spark/Hadoop is really so damn inefficient that something like DuckDB shows where that overhead is

    Criticizes · Oct 2026 · in “MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark”

14. Companies cargo culting Big Data stacks when their data is nowhere near big is very prevalent.

    Criticizes · Apr 2026 · in “I run multiple $10K MRR companies on a $20/month tech stack”

15. To this day, Spark only supports Scala 2.13 although Scala 3 has been around for years now.

    Criticizes · Nov 2025 · in “The Origins of Scala (2009)”

16. Spark is such a headache to run that people would rather pay Databricks to do it for them.

    Criticizes · Oct 2026 · in “MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark”

17. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!"

    Criticizes · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

18. Only once the datasets reach the 100GiB to TiB range is trino/spark faster.

    Criticizes · Aug 2026 · in “AWS Acquires DuckLabs”

19. The second biggest use is Apache Spark/Kafka, but with LLMs now, you can pretty much re implement all of that functionality in pure C.

    Mixed · Aug 2026 · in “Oracle bans AI-generated code from OpenJDK”

20. Had always hoped for something like this since the days of Spark and Frameless.

    Mixed · Mar 2026 · in “Dataframe 1.0.0.0”

21. Spark and Hadoop have their place, maybe not in rinky dink startup land

    Recommends · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”

22. Sounds a lot like Spark before it became so enterprise-focused

    Mixed · Aug 2026 · in “Things I want in a modern relational query language”

23. it's consistently beaten distributed engines like Spark

    Mixed · Nov 2026 · in “Show HN: Duckstring – Pull-orchestrated data pipelines on DuckDB”

24. In fact if you can avoid Akka and Spark, do.

    Mixed · Jul 2026 · in “Show HN: Scala Tutorials – interactive Scala 3 lessons in the browser”

25. go find yourself a trino spark pipeline

    Mixed · Mar 2026 · in “We give every user SQL access to a shared ClickHouse cluster”

26. I realized that it was slow when you used python libraries in the UDFs rather than pySpark's own functions.

    Criticizes · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

27. open source Spark has been taken over, corralled and tammed by the corporate workings and money trains

    Criticizes · May 2026

28. positively ancient distributed JVM-based compute framework running on a single node

    Criticizes · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

29. users can seriously ask themselves if Spark will actually stay relevant that long

    Criticizes · Jul 2026 · in “Show HN: Scala Tutorials – interactive Scala 3 lessons in the browser”

30. I'm personally convinced Spark was the worst thing that happened to the ecosystem

    Criticizes · Nov 2025 · in “The Origins of Scala (2009)”

31. Spark bares some blame here for broken and incomplete documentation.

    Criticizes · Mar 2026 · in “Ask HN: Where have you found the coding limits of current models?”

32. Yes using Python UDFs within Spark pipelines are a hog!

    Criticizes · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

33. one shouldn't really be using vanilla Spark anyways

    Criticizes · Nov 2025 · in “650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark”

34. it's just important that we can say we're using it

    Criticizes · Jan 2026 · in “Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)”

35. Spark solves a problem that does not exist anymore

    Criticizes · Dec 2025 · in “Python Data Science Handbook”

36. a team spending days setting up a Spark cluster

    Criticizes · Jan 2026 · in “Ask HN: When has a "dumb" solution beaten a sophisticated one for you?”

37. Spark's cold start makes that loop impractical.

    Criticizes · Mar 2026 · in “Agent Skills: One-Shot PySpark from the CLI”

38. Way bigger learning curve than just writing SQL

    Criticizes · Sep 2026 · in “I've operated petabyte-scale ClickHouse clusters for 5 years”

39. updating the outdated Scala Spark framework

    Criticizes · Nov 2025 · in “We should all be using dependency cooldowns”

40. much less of a headache to run

    Criticizes · Jan 2026 · in “Why DuckDB is my first choice for data processing”