Apache Kafka

“From a performance POV, Kafka scales horizontally, and appends only need negligible locking.”

“a startup that uses Cloudflare for everything except for Kafka”

“you let Kafka hold onto new data until batch insertions can catch up”

“Kafka clients for several languages are pretty much unusable due to bugs and/or performance”

“Kafka's apis are full of landmines and gotchas, and you can do much better from a UX perspective”

“Kafka I will grant you is a lot to operate, but I would not choose Kafka unless I have a Kafka-shaped problem”

“a benefit of using well known software like Kafka is a lot of problems you will run into have solutions/guidance already available”

“made Twitter stable under the load of several hundred million users”

“The second biggest use is Apache Spark/Kafka, but with LLMs now, you can pretty much re implement all of that functionality in pure C.”

“systems like Kafka give you practical implementations”

“Kafka is actually distributed tech. You can lose nodes and keep operating.”

“anything else a Kafka-like comes with out of the box”

“they build world class software that backs massive systems like elastic search, Kafka”

“OSS projects that scale well, say Linux, Postgres, Kafka, redis”

“Kafka is... a lot. If you need what it provides, then it's great.”

“It's very good at what it does, but painful to manage and usually not worth the pain for anyone who's not in a mega enterprise.”

“We built TBMQ on Kafka because it's battle-tested at scale across the industry”

“it made the immutable approach easier than the spaghetti alternative”

“very fast to start up (~100ms) and use relatively little memory”

“a pretty fun high-level summary of the underlying idea behind them”

“doubling kafka provisioned capacity is simpler, easier, cheaper, and more reliable approach”

“it has caught up fairly well to these alternatives re: memory and startup time”

“Some poor quality software with bad performance, but an established piece of tech regardless.”

“Kafka is taking an interesting turn however with the KIP-932 "Queues for Kafka" initiative”

“An ecosystem building on Kafka clients libraries with various non-Kafka servers would work fine too, but we didn't figure out how to easily do that.”

“Kafka for head of line (HoL) blocking under concurrent events can be an issue though”

“A Kafka consumer does a lot of work coordinating distributed clients in a group, managing the current offset, balancing the readers across partitions”

“If you wanted something fully durable with a stronger persistence story you'd probably use Kafka anyhow.”

“Rewriting the code at that point may be less costly than operating Kafka in the interim.”

“Kafka already solves this problem and gives me message durability, near infinite scale out”

“devs trip over Kafka topics, partitions and offsets when their throughput is low enough”

“Kafka (the messaging/streaming platform) has finally found its natural home.”

“portfolio is starting to look interesting again, especially if it includes Kafka”

“Kafka is a very resource-heavy application that's worth replacing with leaner options”

“Kafka has that out of the box. Where kafka starts to come apart is how to set it up.”

“operational overhead is much lower than running a full Kafka stack”

“It's the very architecture of Kafka itself due to the notorious head of line problem”

“Scaling issues can develop and we in Ops catch flak for your bad design choices.”

“One of the secret traps of Kafka is that it tried to be a collection of nice primitives without that much built on top”

“Going through Kafka allows for reply and setting up multiple independent consumers for the same change event stream.”