So I was eager to know like these AI/ML specific work needs to store something in the persistent kv database? like for example do they needs to use LMDB for any work? Lets say if I have to design a persistent kv database in 2026 for AL/ML workload what should I keep in mind ? and at the first place do they even need ?
If you want to run BM25 ranking or vector search on data lakes (over remote data), you usually have to move or copy that data into a search engine or a dedicated database.
I've prepared a short demo on how you can search over remote data directly from SQL.
For context:
I'm working on a Postgres-compatible search-OLAP database called SereneDB and we've just recently pushed this "Zero-ETL" feature to our repo and are looking for feedback!
Specifically, I'm curious:
So as we know, databases like DragonflyDB can persist data on disk and also use modern async IO techniques like io_uring.
Then why would someone choose a persistent key-value database/storage engine like FoundationDB, TiKV, ScyllaDB, or LMDB-style systems instead?
What architectural or workload differences make those systems preferable over something like DragonflyDB with persistence enabled?
Trying to understand the deeper storage-engine tradeoffs here.
I worked on quant infra for two years. Two things drove me crazy:
I tried the obvious alternatives before building anything.
ClickHouse is great for analytics, but it doesn't have ASOF JOIN. If you've never used ASOF JOIN, it's the SQL operator that lets you do tick-by-tick correlation across feeds — joining a trade with the most recent quote at or before its timestamp. You can fake it with correlated subqueries but it's slow and ugly.
InfluxDB chokes above ~500K events/sec per series. TimescaleDB is fine for slower workloads but not for tick data.
So I started writing my own thing in C++ on weekends. It became ZeptoDB.
**What it does**
- Standard SQL with ASOF JOIN, Window JOIN, xbar (kdb+-style time bucketing), VWAP, EMA — the financial functions you actually use
- 5.52M ticks/sec sustained single-node ingest (8 cores, x86)
- 272µs filter on 1M rows, 248µs GROUP BY
- FIX (350ns), NASDAQ ITCH (250ns), Kafka, MQTT, OPC-UA native consumers
- Python zero-copy bridge — DataFrame in, DataFrame out, no serialization
- Source-available (BSL-1.1, becomes Apache-2.0 in 2030), self-host, K8s Helm chart included
- x86 and ARM/Graviton both supported (test matrix runs on both)
**What surprised me building it**
The wins came from places I didn't expect.
- Highway SIMD on window aggregates: 11x over scalar
- LLVM JIT on filter predicates kept us within kdb+'s range on most queries
- Per-(table, symbol, hour) partition keys gave 2–50x speedup on multi-table workloads. We started with a symbol-only key and it caused weird cross-table data leaks until we found it.
The thing that took longest wasn't performance. It was distributed cluster correctness — split-brain defense, FencingToken in the RPC header, K8s Lease integration, online partition rebalancing. Tick data needs strong correctness guarantees and most of the engineering effort went there, not into making queries fast.
What it's not (yet)
Things I'd rather you know up front than hit in production:
- No JDBC/ODBC drivers. Tableau works through a ClickHouse protocol shim, Excel doesn't.
- No managed cloud. Self-host only for now.
- Window functions over virtual tables aren't supported.
- One query (VWAP 1M p50) has a ~7% gap vs my best baseline due to a clang register-spill issue. Documented in the devlog if you care.
Where it ended up
Started for quants. The same engine now runs in semiconductor fabs (10kHz OPC-UA sensor data), game backends (Kafka telemetry, anti-cheat analytics), and physical AI sensor fusion (ASOF JOIN across LiDAR + camera + IMU). Different verticals, same workload shape.
Happy to answer questions — the kdb+ comparison, why C++ over Rust, why I didn't just put q on top of a free DB, anything.
GitHub: https://github.com/ZeptoDB/ZeptoDB
Site: https://zeptodb.com
I specifically want implementation-focused coding resources for building a MySQL-like database from scratch. I want to actually code things like a SQL parser, query execution engine, storage engine, B+ tree indexes, transactions/MVCC, WAL/recovery, and maybe even a basic optimizer or replication system. I’m searching for GitHub projects, “build your own database” repos, blog series with step-by-step implementations, source-code walkthroughs, or educational mini database engines. Preferred languages are Python. If anyone knows high-quality implementation-focused resources or projects that helped them understand how real databases are built internally, please share.
I wrote about how Log-Structured Merge Trees actually work.
It goes through the write path from WAL → memtable → SSTables → compaction, and covers why LSMs trade read amplification and write amplification the way they do. I also look at leveled vs tiered compaction, skip lists, and Bloom filters, with examples from RocksDB and LevelDB.
I wrote it because a lot of LSM explanations stop at “good for writes,” but that doesn’t help much when you want to understand what the engine is actually doing.
Would appreciate corrections or feedback from people who’ve worked on storage engines.