r/apachekafka

I posted this over on LI last week because I've seen a lot of solo AI-assisted projects here recently. Cool to see people building their own tools but I think the best part of OSS is people actually collaborating and building together rather than working on our own thing in a corner.

To avoid just being a grump though I thought I'd shout out just a few great human-made projects and blogs I like, many of which I came across here:

• ShadowTraffic brilliant tool from Michael Drogalis for rapidly simulating production traffic, he's been building this in the open and with the community.
• Fresha Data Engineering the Fresha team are doing some ground-breaking stuff, a shout out to Nicoleta in particular who's done some great work on Fluss in prod.
• Leo Delmouly's Medium, Leo has done a great series on Kafka + Iceberg which helped me understand why this is such a hard problem.
• Michael Maison's monthly Kafka digest which is my go to resource for understanding what's coming up for the Apache Kafka project.

Disclosure: I used AI assistance to edit this text and code. The implementation and production experience behind it are mine.

This post came from this recent discussion: https://www.reddit.com/r/apachekafka/comments/1vhv9nn/can_kafka_replace_redis_for_cache_synchronization/; as initial post was about k8s / sping-boot stack here I also operate with them.

The question was whether Kafka can replace Redis for synchronizing a cache across about 25 Spring Boot pods. I have used this pattern in production, and the answer is yes for some types of data, but there are a few details that are easy to miss.

The basic setup is simple. Put the data in a compacted topic and keep a local map in every pod. The Kafka record key is the cache key and the value is the latest version of the configuration.

Each pod needs its own consumer group. If all pods use the same group, Kafka distributes the partitions between them and each pod receives only part of the data. With a separate group per pod, every pod consumes the complete topic and builds its own copy.

For a new group use auto.offset.reset=earliest. On every restart the pod replays the topic and recreates the cache. This is why the topic should be compacted and why I would use this only for a small amount of configuration or reference data. It is a good fit for feature flags, routing rules, or tenant settings. It is not a good fit for a large dataset that must be copied into the heap of every pod.

Updates are just records with the same key and a new value. Deletes need a little more care.

The usual pattern is a tombstone: publish the key with a null value. When the consumer sees it, it removes that key from the local map. Kafka keeps the tombstone for some time and later compacts it away together with older values for the same key.

Also, a compacted topic should not be treated as a clean snapshot with one record per key. Compaction runs in the background. During replay, a pod can still read several old values before it reaches the latest one. The consumer has to apply the log in order and let later values replace earlier ones.

The most difficult part is startup.

A pod must not serve requests, run scheduled jobs, consume other queues, or make decisions based on the cache until it has consumed the existing topic. Being assigned Kafka partitions does not mean that the cache is ready. At that moment it may still be empty.

In my example, when partitions are assigned, I capture the consumer's current position and the end offset for every partition. Those end offsets become fixed startup targets. After the listener successfully applies a record to the local map, it advances the processed position for that partition. The pod becomes ready only when every assigned partition reaches its captured target.

The targets must be fixed. If they were read continuously, producers writing new records could keep moving the end offsets while the pod is starting. With an assignment-time snapshot, startup has a finish line. Records written later are still consumed normally, but they do not extend the initial replay.

Progress must be updated after the listener succeeds, not before it runs. Otherwise a failed listener can be counted as processed even though the cache was not updated. Progress and errors also need to be tracked per partition, and assignments need to be recalculated after a rebalance. Empty partitions are already complete when their current position equals their target.

Kubernetes readiness solves only the HTTP part. A pod that is not receiving web traffic can still run scheduled methods or start another message listener. Those also need to wait for the same replay status. In the example I expose the status through a Spring Boot readiness health indicator and use a small annotation to prevent scheduled methods from running before the replay completes.

After startup, the caches are eventually consistent. One pod can briefly have an older value due to lag, a rebalance, or an outage. That is acceptable for some configuration, but not for balances, inventory, permissions, or anything else that must change everywhere at the same time.

I put the complete Spring Boot and Docker Compose example here: https://github.com/javaAndScriptDeveloper/kafka-backed-local-read-replica-article

It includes the compacted topic, one consumer group per application instance, tombstone handling, startup replay tracking, readiness integration, and gating for scheduled jobs. I would be interested to hear how others prevent work from starting while a local Kafka-backed cache is still being rebuilt.

Hey all! I’ve worked with Kafka here and there over the past few years and I’ve been really frustrated with the tooling.

I took a stab at a tool which simplifies reads and writes to topics down to:

frogo get <topic> —from <offset-like> —to <offset-like>
frogo put <topic> —file <file-w-one-msg-per-line>

GitHub link: frogo-cli

As a brief overview:

For ‘frogo get’ - the main idea is that an ‘offset-like’ supports literal offsets, timestamps, dates, and aliases (START, END, FUTURE).

For ‘frogo put’ - the main idea is you have a file which has one message per line. Multiple formats are supported (e.g. base64 for binary data)

Some other features I’ve added:
- mockserver: thin wrapper exposing a franz-go mock server (frogo mockserver)
- configuration profiles (use —profile or FROGO_PROFILE)
- multiple input / output formats (—format)
- fixture topics with example data (frogo topic demo)

This tool is by no means comprehensive, and I haven’t added support for things like:
- consumer groups
- schemaregistry
- certain authN / authZ configs

But… I hope this could serve as a helpful development / ops tool for those not needing all the bells and whistles.

Any feedback / criticism would be much appreciated. Feel free to create an issue on the GitHub for any feature requests!