u/vira28

Recently, I had a discussion with a database kernel engineer about implementing inverted on Iceberg tables. Coming from OLTP background, Index is typically a secondary data structure (like B-Tree) and i never thought of using tables itself as an index.

So, I implemented a simple version of an inverted index in my project Streambed (Postgres for analytics using DuckDB) using DuckLake.

Here is the implementation https://github.com/viggy28/streambed/issues/101

I had to fork DuckLake since it doesn't support external hooks for bringing custom predicates. Here is the upstream issue https://github.com/duckdb/ducklake/issues/1498

I thought folks in this subreddit might be interested, and I also would love to hear if there is any feedback.

I've been building an open-source (Apache 2.0) "analytics for Postgres" project called Streambed, and the piece this subreddit might find interesting is the DuckDB/DuckLake layer.

Streambed streams Postgres WAL changes into lake storage and serves queries through DuckDB as the query engine.

I started Iceberg as the default target but one of the community members recently mentioned about DuckLake. It caught my attention immediately (since it uses Catalog as the DB). So I added the DuckLake support and did the benchmark.

Ducklake is 41 times faster than Iceberg in my benchmark.

Benchmark slice: 1M rows, 100k updates, flush=1,000

  Write path:
  Iceberg COW:        269s
  Iceberg MoR:        4.7s
  DuckLake + DuckDB:  6.6s

  Full aggregate median:
  Iceberg MoR: 87s
  DuckLake:    16ms

  TopN median:
  Iceberg MoR: 85s
  DuckLake:    21ms

Takeaway: for a DuckDB-centered CDC path, DuckLake looks like the cleaner default. Iceberg is still great when broad multi-engine interoperability matters.

DuckDB/DuckLake is a magical primitive. So much to build on top of it.

Blog: https://streambed.dev/blog/ducklake-target-support/

Benchmark: https://github.com/viggy28/streambed/blob/main/docs/benchmarks/ducklake-vs-iceberg.md

Anyone who works on a production Postgres knows the feeling. Every command you run, you're walking a tightrope. One typo, one wrong terminal tab, one bug in the app that turned a filter into a full-table query, and now you're doing PITR or restoring from backup at 3am.

I've spent years as a DBA in charge of critical production workloads. Most of the time the rope holds. Sometimes it doesn't.

pg_savior is a Postgres extension that hooks the planner and refuses the obvious dangerous shapes:

• DELETE / UPDATE without a WHERE
• CREATE INDEX without CONCURRENTLY
• DROP DATABASE
• ALTER COLUMN TYPE that triggers a full rewrite
• DELETE WHERE id > 0 (planner row estimate gives intent away)

When you really mean it: SET LOCAL pg_savior.bypass = on for the transaction, and the guard steps aside.

It's an extension, not a proxy — psql against a local socket, ORMs, migration tools, cron jobs, AI agents with DB credentials all hit the same hook. Nothing routes around it.

Three hooks do the work: post_parse_analyze_hook, ExecutorStart_hook, ProcessUtility_hook.

• Code: https://github.com/viggy28/pg_savior
• Design notes: https://viggy28.dev/article/pg-savior-seatbelt-for-postgres/
• PGXN: https://pgxn.org/dist/pg_savior/

What other dangerous queries should pg_savior catch? Also, curious if you have best practices to catch these mistakes.