u/CautiousUse8597

More information about Lakebase and LTAP

Databricks spent most of the Summit keynote telling us LTAP means "no more pipelines, no more ETL, one copy of data." Fine. I've heard "no more ETL" enough times to be suspicious of it on reflex. But I got curious about the one part nobody really spells out in the press releases: if your app is writing plain Postgres rows, how does that same data show up in Iceberg as columns, fast enough to query, without a pipeline you can actually see? So I went reading. Here's the mechanism as best I can piece it together.

First, why this is even a problem. Postgres stores data by row. Everything about one record sits together on disk, which is exactly what you want for "give me user 48213 and update their balance", since you touch one row and you're done. Analytics wants the opposite. "Average order value over the last 90 days" only needs one column out of forty, but in a row store you still drag every row (and every other column) off disk to get at it. Columnar formats like Parquet flip the layout so each column is stored together:

Row store (Postgres):

[id=1, name=Ana, amount=50]

[id=2, name=Ben, amount=80]

[id=3, name=Cy, amount=20]

Column store (Parquet / Iceberg):

id: 1, 2, 3

name: Ana, Ben, Cy

amount: 50, 80, 20

Now "sum the amounts" reads one tidy contiguous list and skips everything else. It also compresses far better, because similar values end up sitting next to each other. Old Lakebase basically kept Postgres data in Postgres format on object storage, so you still needed a conversion step before the analytical engines could do anything useful with it. LTAP's whole pitch is killing that step.

The thing actually doing the work is Moonlink, a component from the Mooncake team Databricks bought last year. It's a replication engine (written in Rust, for what it's worth). It taps Postgres's logical replication stream, the same change feed Postgres already emits for replicas. Every insert, update and delete flows out of that feed. Moonlink consumes it and mirrors the changes into Iceberg, but rewritten as columns, with sub-second lag.

The detail I found genuinely clever is where the row-to-column conversion happens. Object storage (S3 and friends) is slow: response times in the seconds, way too slow to serve actual transactions. So Postgres keeps a fast caching tier in front of it, and that tier usually has spare CPU sitting idle. Databricks does the transcode right there, on that idle CPU, before the data ever lands in object storage. And because going row-to-column compresses something like 10x, you've also shrunk what you have to push down to S3 in the first place. You're not paying for a separate conversion job later; you're paying with cycles that were going to waste anyway, and you ship less data for the trouble. (Reynold Xin walked through this in a VentureBeat interview if you want the source.)

The other half is the freshness trick, because there's normally an annoying tradeoff here. If Moonlink wrote every single change straight into a new Iceberg file, you'd get freshness but also a blizzard of tiny files and constant metadata commits, which is miserable to live with. If it batches writes up to be efficient instead, your analytics goes stale. Moonlink dodges this with what they call union reads. Newly arrived rows sit in an in-memory columnar buffer (Arrow). A query then reads the committed Parquet files on object storage, that in-memory buffer, and any pending updates or deletes, all stitched together as one logical table. That means an analytical query can see data that hasn't even been written into an Iceberg snapshot yet. That's how they claim sub-second freshness without drowning in small files.

One thing worth being clear-eyed about: this isn't HTAP in the old "one engine does both" sense, no matter how the slides read. Postgres is still the transactional engine; Spark/Photon and the new Reyden engine are still the analytical ones. A commit in Postgres is not running your analytical query inside the same transaction. What's actually unified is the storage. One logical copy of the data, one write path (Postgres to Moonlink to Iceberg), and several read paths sitting on top. "One copy for everything", not "one engine for everything". Which, honestly, is probably the more achievable version of the dream anyway.

Caveat: LTAP was just announced and is rolling out as part of Lakebase, so most of this is stitched together from the announcement, interviews, and a couple of good technical breakdowns rather than me running it in prod. If anyone's got it live in their workspace and I've got a detail wrong, please correct me.

TL;DR: Your app writes rows to Postgres like normal. Moonlink tails Postgres's replication stream, converts those changes to columnar Parquet on idle CPU in the caching tier (which also compresses around 10x before anything hits S3), and writes them into Iceberg. Queries read the committed columnar files plus an in-memory buffer of brand-new rows together (union reads), so analytics stays fresh within a second. Separate engines, one shared copy of the data, not classic single-engine HTAP.