186 database records. 187 Kafka events. Where would you look first?
I’ve been experimenting with turning distributed-systems failure modes into fictional incident investigations.
In this one, a lab registers 186 samples. Registration succeeds in the database, then publishes label-print commands to Kafka.
Later the team discovers 187 print events.
Eventually they reconstruct the sequence:
DB commit succeeds → Kafka publish succeeds → ACK is lost → application retries → duplicate print command → duplicate physical label → subsequent labels shift by one.
I used producer idempotence + business-level deduplication on label ID as part of the remediation, with a scanner guardrail at the physical boundary.
Architecture question: Would you consider that sufficient, or would you redesign the DB→Kafka boundary around an outbox/CDC approach?