Tech Review: DuckLake - From Parquet to Powerhouse
u/empty_cities — 5 days ago
▲ 17 r/DuckDB
Understanding DuckLake's Sorted Tables Feature
https://thefulldatastack.substack.com/p/understanding-ducklakes-sorted-tables A sponsored post about DuckLake's Sorted Tables feature. It allows you to specify a sort config for a table so that unsorted data will automatically get sorted in a certain way with inserts, flushing and compaction.
For queries ran regularly on high cardinality columns like id or timestamp this can optimize reads. When data is sorted physically in Parquets it allows for both file skip and row group skip to only get the data you need for the query (a.k.a predicate pushdown).
I made a high level mental model image here I thought came out well to explain the file skip and predicate pushdown (row group skip).
u/empty_cities — 11 days ago
▲ 10 r/DuckDB
Understanding DuckLake's Inlining Feature
https://thefulldatastack.substack.com/p/understanding-ducklakes-inlining
This is a sponsored article I wrote doing a deep dive into DuckLake's inlining feature. I was really happy with how it came out.
Inlining is how DuckLake solves the "Small File Problem" that nags lakehouse designs. When you do an INSERT into DuckLake it will assess how many rows are in the insert. If the number of rows is above its default threshold of 10, then it will write the insert to a Parquet file.
If it is fewer rows than the threshold, it will inline the data into the metadata catalog (Postgres, DuckDB or SQLite). The data will be stored there until you flush it to Parquet manually.
This allows you to stream to a DuckLake with single events at a very high frequency without worrying about tens of thousands of data files being created every day.
u/empty_cities — 19 days ago
DuckDB Basics: Reading and Importing Data
u/empty_cities — 2 days ago