Apache Parquet

“Parquet is also 13 years old and not the GOAT at compression”

“it's a great combination for data exploration tools that run in the browser”

“with their built-in columnar compression will result in more performance AND space savings”

“Wouldn't Avro or Parquet be solid choices for something like this”

“An LSM-style layer over Parquet gets all of this fairly naturally”

“but with parquet you need large row group batches. Worst case (low log volumes and a time-based flush) you’d end up with loads of tiny row groups.”

“Columnar, compression, fast, succinct. All good things.”

“you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens”

“Sadly Parquet is useless for the append-only layer.”

“if you are just doing pure analytics and nothing else, Parquet will probably continue to do the job for you just fine”

“my main gripe with parquet (single table per file) is not even addressed”

“DuckDB can do all manner of nifty optimizations while reading its own native format or Parquet”

“a dead-simple, heavily documented byte specification like Parquet”

“A big problem with parquet, which this aims to replace, is that it's hard to add new encodings”

“I can get the zstd parquet to be up to 10x (or more) smaller than an equivalent Postgres table”

“You’ll see vastly smaller file sizes (even more so if you use zstd compression)”

“only compressing the parquets in LZ4 and getting a fuck ton of more space savings”

“parquet really helped with interop”

“It should result in a smaller file size due to columnar compression.”

“there really aren't any good pure Swift libraries for Arrow or Parquet”

“I think this would be pretty straightforward for Parquet with ZSTD compression”

“Just Parquet files + simple partitioning + commits on Hugging Face.”

“I chose Parquet because it is columnar and compressed”

“Plus isn't the least wasteful format”

“Parquet might win for table scans depending on the underlying storage medium”

“I think parquet and arrow are great formats”

“Depending on your needs, the right tool might be Parquet or Arrow or protobuf or Cap’n Proto”

“while Parquet fixes the "delimiter problem", it doesn't fix the "encoding problem"”

“Pulling in Parquet and all of its dependencies is utter overkill.”

“Parquet and Feather can have excellent performance and space efficiency if you use them for their intended purposes”

“The heavy dependency footprint of parquet-java was the main driver for kicking off this project.”

“this is significantly faster than parquet-java for the same task”

“extremely unpleasant to use with its massive fan-out of dependencies”

“parquet has fantastic support and is still a pretty good choice”

“Creating the Parquet data is a pain since it's not append-able”

“Parquet with row groups is fast and compresses well.”

“Parquet, Iceberg, and other open formats seem good”

“not only do you get seekable data, you can fetch only the columns you want too”

“Standard practice is to use formats like WebDataset, Parquet, or TFRecord to chunk small files into large, sequential blobs.”

“compression better with Parquet”