u/Low_Brilliant_2597

Deletion Vectors are among the major improvements introduced in Apache Iceberg v3. Instead of creating separate position delete files, a deletion vector stores a compact bitmap for a single data file that marks which row positions have been deleted.

This approach improves performance and preserves more useful statistics for data files. It also makes it easier to compare previous and current deletes, simplifying the processing of a table’s row-level changes as a stream.

Every database design is a tradeoff between performance, correctness, and complexity.

- Storage engines trade off reads, writes, and space.

- Indexes trade update cost for faster lookups.

- Optimizers trade planning time for execution speed.

- Concurrency control trades parallelism for correctness.

- Recovery trades write overhead for durability.

- Distributed database systems trade latency, consistency, and availability.

From the first DBMS to the relational model, SQL, transaction processing, and enterprise databases like Oracle and PostgreSQL, these pioneers laid the foundation for database systems.

There are also many other database researchers and engineers who have contributed a lot to database systems, including Patricia Selinger, Raymond Boyce, and many others.

Object storage is becoming the primary storage layer for modern database systems. Not as backup, archival storage, or a cold tier, but as the primary durable storage layer. As a result, more and more data systems are being designed around the assumption that durable data lives in object storage, while compute becomes ephemeral and elastic.

Examples span almost every category of the data stack: Snowflake and Databricks, Neon and TiDB, RisingWave streaming database, turbopuffer, LanceDB, Milvus, Chroma, and SlateDB.

The workloads, query engines, and consistency models are different, yet they all converge on the same architectural principle: decoupling compute and storage, with object storage serving as the durable storage layer.

The benefits of using object storage are obvious: low storage cost, unlimited capacity, high durability, elastic compute, and simplified operations. But object storage is not a free lunch. Its primary challenges are no longer disk management or capacity planning, as in traditional storage systems. Instead, they include request amplification, metadata scalability, read/write operation costs, cache management, tail latency, and consistency semantics.

As all data systems have their own trade-offs and related engineering challenges, so do object-storage-native database systems. Still, this is becoming a fundamental storage architecture pattern used across databases with different workloads.

For more than 50 years, people have been predicting the death of SQL. As Prof. Andy Pavlo says, "Somebody invents a SQL replacement every decade. It then fails and/or SQL absorbs the key ideas into the standard. Every NoSQL DBMS (except Redis) now supports SQL."

Today, the latest challenge comes from AI. At a recent keynote, Databricks CEO Ali Ghodsi declared that "AGI is here today." The implication is that natural language interfaces and agents may eventually replace SQL. Yet the reality is more nuanced.

The recently released BEAVER enterprise Text-to-SQL benchmark shows that even the most advanced models still struggle with real-world enterprise SQL tasks. As of 27 February 2026, the latest benchmark results are: Claude 4.5 Sonnet: 11.4% and GPT-5.2: 10.8%.

These models have improved a lot over the past few months. But when faced with large enterprise schemas, complex joins, domain-specific knowledge, nested queries, and analytical workloads, they are still far from reliably replacing SQL expertise.

The lesson is that SQL keeps proving remarkably resilient. Just as it survived many "SQL killers," it is likely to evolve alongside AI rather than be replaced by it. After more than half a century, SQL remains the lingua franca of data. RIP SQL? Not anytime soon.

The RUM Conjecture explains one of the most important realities of database design. There is no perfect storage engine. All databases make different tradeoffs. And those tradeoffs can be explained through a simple framework: The RUM Conjecture.

RUM stands for: Read efficiency, Update (write) efficiency, Memory (space) efficiency. The core idea is simple: You can optimize for any two. But you can not simultaneously optimize all three. Storage engine designers focus on what they are willing to sacrifice. Every database storage architecture sits somewhere inside the RUM triangle.

For example, B+ Trees favor fast reads and space efficiency while paying for higher write costs and random I/O. Pure log structures favor fast writes and sequential I/O while paying for expensive reads and space amplification. Hash tables favor fast reads and fast writes while paying for large memory consumption and poor space efficiency.

LSM Trees take a more balanced approach: Write to MemTable → Flush to SSTables → Compact later. In other words: Optimize writes first. Organize data later. This allows LSM-based systems to achieve high write throughput, good read performance, and reasonable space efficiency while accepting compaction overhead, read amplification, and write amplification.

The interesting part is that the hard problem isn't optimization itself. It's understanding the cost of optimization. Improve reads? More indexes, more metadata, more maintenance. Improve writes? More files, more versions, more read work later. Improve space efficiency? Less redundancy, less metadata, slower lookups. Every improvement introduces a new cost somewhere else in the system. As the RUM Conjecture teaches: Pull one rope, and the other two pull back.

Modern storage engines are ultimately exercises in balancing read amplification, write amplification, and space amplification rather than eliminating them entirely. This is also why LSM Trees became the dominant storage architecture behind many data systems. Not because they maximize one dimension. But because they strike a practical balance across all three. Understanding the RUM Conjecture is ultimately about understanding a fundamental truth of system design: Every storage engine is a tradeoff. The question is not whether tradeoffs exist. The question is, which tradeoffs best fit your workload.

LSM Trees power many of today's databases. Because most databases face a fundamental challenge: how to support massive write throughput without constantly rewriting data.

Traditional B-Tree systems update data in place: Find page → Modify page → Write page back. While LSM Trees take a different approach: Write to MemTable → Flush to SSTables → Compact later. In other words, write fast now and organize later.

This simple idea optimizes them for high-ingestion workloads.

But every design comes with tradeoffs. LSM-based systems gain fast writes, sequential I/O, and scalability, while paying for compaction, write amplification, and read amplification.

So, understanding LSM Trees today is not just about storage engines. It's about understanding one of the foundational building blocks behind most of today's databases.

Hey, I want to share this blog about the journey and lessons learned from RisingWave, a streaming database, that migrated from C++ to Rust by deleting 276k lines of code, discarding 7 months of development, and starting from scratch. I really hope this will be helpful for anyone working on or starting a data infra project with Rust and want to understand its pros and cons.

* I work at RisingWave.*