[Design Help] Efficient key-based lookup on a large Kafka topic for a background verification workflow
We are building a background workflow where for a given input, we need to find the corresponding message in Kafka and verify some fields on it.
Our Kafka setup:
- compacted topic, 24 partitions, ~200M messages per partition (~2.5B unique keys total)
- ~700 bytes per message, so roughly 1.75TB of data
The lookup pattern is key-based, ~10k/sec, background process so some latency is fine.
We do have a way to derive the partition from the key and an API to get the offset, so seek+fetch is technically possible — but our Kafka brokers are a shared resource across teams and we don't want to hammer them with random-access reads at this scale.
How would you build the lookup layer here? What would you use, how would you keep it in sync with the topic, and anything to watch out for at this scale?
For context, we're leaning towards RocksDB — consuming the topic, storing only the fields we need for verification, and using Protobuf to keep it compact. But curious if there are better approaches or gotchas we are not thinking about.