Building a Key-Value Database on S3 in Go
▲ 32 r/databasedevelopment+2 crossposts

Building a Key-Value Database on S3 in Go

Isledb is an embedded Go key-value database library on S3 using LSM trees.

  1. Writes are batched in the memtable and then flused into SST which gets uploaded to object store.
  2. A CAS conditional(e-tag) commits of Current Manifest is made with fencing for the SST to be made viisible for the reader.
  3. Compaction can run on seperate process.
  4. Reader reads directly from s3 object store

Source Code: https://github.com/ankur-anand/isledb

Blog: Outlines all the design in detail and various nuances and decisions: https://medium.com/@ankur_anand/building-a-key-value-database-on-s3-in-go-fb08d040f0af

medium.com
u/ankur-anand — 8 days ago
▲ 10 r/golang

Using object storage as a WAL replay layer in a Go database

Unisondb is dynamodb inspired multimodal database written in go. Aim at replicating to 1000's of nodes. It started with a simple replication model.

```
writer -> gRPC stream -> replica
```

A writable node owns the WAL. A replica connects over gRPC. The writer streams log records. The replica applies them in order.

We have added a a second replication path inside the Unisondb:

```
writer -> blob store -> replica
```

Project Repo: https://github.com/ankur-anand/unisondb

Looking forward to feedback and suggestions.

unisondb.io
u/ankur-anand — 2 months ago