Login

r/Clickhouse

▲ 13 r/Clickhouse+2 crossposts

If you're consuming from Kafka and writing into ClickHouse, sync inserts at high message rates will hurt you. Async insert mode helps a lot but the buffering and dedupe behavior isn't always obvious.

Wrote this up from our my experience building a stream processing pipeline.

Curious how others are handling the Kafka → ClickHouse write path.

u/Marksfik — 8 days ago
▲ 25 r/Clickhouse+4 crossposts

paradedb/benchmarker: a workload agnostic, multi-backend benchmarking tool.

Hi r/postgresql!

We just open sourced ParadeDB Benchmarker, a multi-backend benchmarking framework built on top of the excellent Grafana k6 (blog post).

One of the goals was avoiding a shared query abstraction layer. PostgreSQL queries stay PostgreSQL queries, with their own driver and native SQL.

Supports PostgreSQL, Elasticsearch, OpenSearch, ClickHouse, MongoDB, and ParadeDB with:

  • mixed read/write workloads
  • support for docker-compose profiles per backend
  • dataset loader
  • config and setup capture
  • live metrics + exported reports

One of the ah-ha moments I had building this was using the pgx Go driver in anger for the first time, I'm a Rust guy, but I'm seriously impressed with pgx and what it can do.

Any comments welcome, we will be using this to benchmark ParadeDB, but you can write your own datasets and workloads which have nothing to do with full-text search.

github.com
u/jamesgresql — 9 days ago