How do you move consumers to a new topic without losing your place?
Every few weeks we have to move consumers from one topic to another. In our case it's pretty much always topic renames (naming convention cleanup). The tricky part is the ordering: producers don't switch to the new topic until consumers are already on it. Our consumer switching over is what triggers them to migrate, and even then they move on their own schedule. So the old topic keeps getting messages for who knows how long after we've switched.
That means we can't do the comfortable thing (stop producers, let consumers drain the old topic, then move everyone). We have to switch consumers to the new topic first, then keep copying whatever still lands on the old topic into the new one. Without losing unread messages and without changing the consumer group.
The obvious answer is a script around kafka-consumer-groups or kcat but that doesn't work for us. In a lot of our environments nobody has shell access to anything that can reach Kafka. Also the copy can run for days depending on when producers decide to switch, and it runs on k8s pods that can restart at any moment. So we ended up writing a small internal service that does the copy, saves its progress somewhere and resumes after restart instead of starting over or duplicating messages. We now also use it to re-send a slice of a topic (say everything after some timestamp) when a consumer needs to reprocess messages.
Is this a normal problem or is our setup weird?
If you've done a migration like this, what did you use and did it actually work?