Using NotebookLM, I turned Reynold Xin's blog into a nice deck. I hope this graphical version is more consumable to some of the folks who prefer to read infographics like myself. Enjoy!
Original blog:
https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage
Read a Databricks piece on Lakebase/LTAP and wrote a short note on the idea that clicked for me: maybe OLTP and OLAP should meet at storage, not inside one engine.
I've been watching cautiously Databricks' announcement of OLTP/OLAP unification from their summit. They dub it Lake-TAP as opposed to HTAP. Now their cofounder published this technical piece that popped up on my HN:
https://www.databricks.com/blog/lakebase-ltap-rethinking-database-storage
This made me look at it very differently from their original announcement at their conference. Basically, I think this is a great capability of their Lakebase Postgres offering (assuming it works, I still haven't had my hands on it): You store your data in Postgres but the source of truth is stored in Iceberg/Parquet on S3. That means you can now do warehousing on all the Postgres data directly because it's just our usual big data stack of Iceberg etc.
This seems like a bigger/different deal than how the company presents it. That is, it's really Postgres on Iceberg more than some grand unification theory. The fact that both are open source standards (Postgres and Iceberg) means it could have legs. Personally, I think CDC is a necessary evil for data engineering. But maybe no longer necessary?
The article argues that LTAP in Lakebase is less about replacing ETL and more about reducing unnecessary synchronization between operational and analytical systems. By sharing a governed data foundation, organizations can simplify architecture, reduce latency, and provide AI systems with fresher context. However, it is presented as an architectural option rather than a universal replacement for existing data platforms, and sound data engineering practices remain essential.
Databricks spent most of the Summit keynote telling us LTAP means "no more pipelines, no more ETL, one copy of data." Fine. I've heard "no more ETL" enough times to be suspicious of it on reflex. But I got curious about the one part nobody really spells out in the press releases: if your app is writing plain Postgres rows, how does that same data show up in Iceberg as columns, fast enough to query, without a pipeline you can actually see? So I went reading. Here's the mechanism as best I can piece it together.
First, why this is even a problem. Postgres stores data by row. Everything about one record sits together on disk, which is exactly what you want for "give me user 48213 and update their balance", since you touch one row and you're done. Analytics wants the opposite. "Average order value over the last 90 days" only needs one column out of forty, but in a row store you still drag every row (and every other column) off disk to get at it. Columnar formats like Parquet flip the layout so each column is stored together:
Row store (Postgres):
[id=1, name=Ana, amount=50]
[id=2, name=Ben, amount=80]
[id=3, name=Cy, amount=20]
Column store (Parquet / Iceberg):
id: 1, 2, 3
name: Ana, Ben, Cy
amount: 50, 80, 20
Now "sum the amounts" reads one tidy contiguous list and skips everything else. It also compresses far better, because similar values end up sitting next to each other. Old Lakebase basically kept Postgres data in Postgres format on object storage, so you still needed a conversion step before the analytical engines could do anything useful with it. LTAP's whole pitch is killing that step.
The thing actually doing the work is Moonlink, a component from the Mooncake team Databricks bought last year. It's a replication engine (written in Rust, for what it's worth). It taps Postgres's logical replication stream, the same change feed Postgres already emits for replicas. Every insert, update and delete flows out of that feed. Moonlink consumes it and mirrors the changes into Iceberg, but rewritten as columns, with sub-second lag.
The detail I found genuinely clever is where the row-to-column conversion happens. Object storage (S3 and friends) is slow: response times in the seconds, way too slow to serve actual transactions. So Postgres keeps a fast caching tier in front of it, and that tier usually has spare CPU sitting idle. Databricks does the transcode right there, on that idle CPU, before the data ever lands in object storage. And because going row-to-column compresses something like 10x, you've also shrunk what you have to push down to S3 in the first place. You're not paying for a separate conversion job later; you're paying with cycles that were going to waste anyway, and you ship less data for the trouble. (Reynold Xin walked through this in a VentureBeat interview if you want the source.)
The other half is the freshness trick, because there's normally an annoying tradeoff here. If Moonlink wrote every single change straight into a new Iceberg file, you'd get freshness but also a blizzard of tiny files and constant metadata commits, which is miserable to live with. If it batches writes up to be efficient instead, your analytics goes stale. Moonlink dodges this with what they call union reads. Newly arrived rows sit in an in-memory columnar buffer (Arrow). A query then reads the committed Parquet files on object storage, that in-memory buffer, and any pending updates or deletes, all stitched together as one logical table. That means an analytical query can see data that hasn't even been written into an Iceberg snapshot yet. That's how they claim sub-second freshness without drowning in small files.
One thing worth being clear-eyed about: this isn't HTAP in the old "one engine does both" sense, no matter how the slides read. Postgres is still the transactional engine; Spark/Photon and the new Reyden engine are still the analytical ones. A commit in Postgres is not running your analytical query inside the same transaction. What's actually unified is the storage. One logical copy of the data, one write path (Postgres to Moonlink to Iceberg), and several read paths sitting on top. "One copy for everything", not "one engine for everything". Which, honestly, is probably the more achievable version of the dream anyway.
Caveat: LTAP was just announced and is rolling out as part of Lakebase, so most of this is stitched together from the announcement, interviews, and a couple of good technical breakdowns rather than me running it in prod. If anyone's got it live in their workspace and I've got a detail wrong, please correct me.
TL;DR: Your app writes rows to Postgres like normal. Moonlink tails Postgres's replication stream, converts those changes to columnar Parquet on idle CPU in the caching tier (which also compresses around 10x before anything hits S3), and writes them into Iceberg. Queries read the committed columnar files plus an in-memory buffer of brand-new rows together (union reads), so analytics stays fresh within a second. Separate engines, one shared copy of the data, not classic single-engine HTAP.
More info here: it seems that no data duplication and CDC pipeline is needed anymore. The same data wold be used for both Trasactional and analytical workflows.
​