u/arintonakos12

Hey r/vectordatabase,

I’m working on DataVo, an embedded relational database engine written in C#. I’d like to get feedback from people who have worked on storage engines, indexing, query execution, or embedded databases.

This project started as a university project in 2023. I’ve been revisiting it as a real research/engineering project because I wanted to explore what an embedded database can look like when it is designed around the .NET runtime instead of sitting behind a native provider boundary.

To be transparent, AI tools helped with boilerplate, docs, test scaffolding, and iteration speed. I still own the design decisions, review the code, run the tests, and maintain the project. I’m not trying to hide that. I’d rather have the work judged on the architecture, tests, benchmarks, and limitations.

The current engine includes:

- SQL-style DDL/DML/query support

- In-memory, disk, and LSM storage modes

- WAL-backed disk and LSM durability modes

- Session-bound transactions

- MVCC snapshot visibility

- Scalar indexes

- INT and GUID index fast paths

- VECTOR(n) columns

- HNSW and flat vector index paths

- SIMD distance kernels

- Roslyn source generators for compiled query paths -> Personally I think this is the most interesting feature of the DB engine, since supported query shapes can move parse/planning/mapper setup ahead of runtime and do less dynamic work when the app is running

- Incremental view maintenance (IVM) / reactive query support -> This is the other big thing i'm exploring that produce very good results as compared to SQLite / DuckDB (however there we are comparing apples to oranges)

- ADO.NET-facing package work

- Early EF integration

The main design goal is a C#-native embedded engine with allocation-aware hot paths. Some internal read/write paths are designed to avoid steady-state allocation, reduce GC pressure, and make performance easier to reason about in local-first .NET apps, game tooling, simulations, and other latency-sensitive workloads.

This is not meant to be a SQLite or Postgres replacement. SQLite is one of the obvious embedded baselines, so I benchmark against it, but the point is not “DataVo beats SQLite overall.” SQLite is mature, extremely reliable, and far ahead on production hardening.

DataVo is alpha software and still has missing features and edge-case bugs.

The benchmark page includes results for several workloads, including flat CRUD, disk CRUD with WAL, thread scaling, YCSB-style mixed reads/writes, vector search, deep document loading, and some comparisons against SQLite, LiteDB, and DuckDB depending on the scenario.

The benchmark results are local/CI measurements, not universal rankings, and the relaxed LSM numbers are not the same durability contract as strict fsync.

https://preview.redd.it/fjb5eut3olbh1.png?width=3997&format=png&auto=webp&s=12dc31b6d8a1dfad5a7db6b701959e5ae9c32a8a

https://preview.redd.it/bxfsxvt3olbh1.png?width=4499&format=png&auto=webp&s=a1b381e6f3b83f1b917ce575377d2d146b26e780

https://preview.redd.it/j96vzut3olbh1.png?width=2774&format=png&auto=webp&s=fc39e586b6743eda8d9484c3f406d0356196c7e6

https://preview.redd.it/4ty0yut3olbh1.png?width=2736&format=png&auto=webp&s=07f51789c8f663456f677217bd8bb167808947e1

I’m mainly looking for feedback on the direction of the project. Does a C#-native embedded database make sense for .NET workloads, or is this solving a problem that people would rather handle with SQLite/Postgres plus libraries?

I’m also interested in what you would want to see next to evaluate it seriously: different benchmarks, durability tests, more SQL coverage, better EF/ADO.NET integration, vector search work, or something else.

Repo: https://github.com/ArintonAkos/DataVo-DBMS

Docs / benchmarks: https://arintonakos.github.io/DataVo-DBMS/manual/performance/benchmarks

Write-up: https://medium.com/p/bfeac66c5cac

I’ve been working on DataVo, an open-source, embedded SQL + vector engine written entirely in C#. There are no native C/C++ binaries to bundle or cross-platform targets to fight with, it’s just a single managed library.

When you build a database engine in .NET, the garbage collector is your ultimate bottleneck. Object allocations on every operation completely destroy your P99 latency due to Gen 0 churn. To get around this, the entire engine is designed for zero steady-state allocations in the hot path.

A few ways it does that:

• The write path is an LSM tree where the MemTable rents 32MB slabs from ArrayPool<byte>.Shared. Rows are serialized directly into the raw byte arrays using a bump allocator, so the GC never even tracks them.
• It uses Roslyn source generators to compile query execution paths at build time. If you annotate a query, it generates a static, typed row reader that maps straight to your CLR type, skipping runtime AST parsing and avoiding object boxing.
• Vector search runs flat and HNSW paths entirely over contiguous managed memory using System.Numerics.Tensors, eliminating the P/Invoke marshaling tax.

I ran the benchmarks on a standard, noisy GitHub Actions Linux runner against SQLite and LiteDB to see how it handles real-world cloud hardware. On a mixed concurrent workload, it hit 2,368,026 ops/sec (SQLite hit ~199k on the same box). For a 10k vector workload, it allocated about 10MB of memory, while LiteDB's brute-force path churned through 208GB of garbage.

To be entirely transparent about where it loses right now: SQLite's native sqlite-vec extension has a tighter C-kernel and still wins on single-query vector latency (3.1ms vs 10.5ms). Also, my current HNSW graph construction is single-threaded and brutally slow, taking 160 seconds to build an index that SQLite finishes in under a second. That's the next big optimization target.

The repo and deep-dive article are below. I'd love to get your thoughts on the architecture or have you guys try to break it.

Full Write-up: https://medium.com/@arintonakos/bypassing-the-net-gc-how-i-hit-2-3-million-ops-with-a-c-native-embedded-database-bfeac66c5cac

GitHub Repo: https://github.com/ArintonAkos/DataVo-DBMS