u/No_Many1887

Hi everyone,

Basic SQL constraints (NOT NULL, foreign keys, data types) catch syntax errors, but subtle semantic bugs often slip right past them (e.g., currency swapped from USD to EUR without warning, dropped decimal places, or inverted credit/debit signs).

What is the most insidious data corruption you’ve encountered in transactional data that passed all basic schema checks? How did your team eventually detect it?

Hey folks,

If an ingestion batch of 1,000 payment records arrives where 995 rows are completely valid and 5 rows have missing critical fields (like receiver_account is NULL):

Do your pipelines usually:

1. Reject/Hold the entire batch to preserve batch atomicity and ordering?
2. Accept the 995 valid rows and route the 5 bad rows to a Dead-Letter Queue (DLQ)?

What are the trade-offs you run into with transactional integrity versus keeping pipeline throughput moving

I’m trying to understand how people decide whether messy data can be safely repaired or whether it should be left alone/escalated.

For example, suppose you discover:
missing values
duplicates
invalid-looking values
inconsistent formats
unusual/outlier values

Some of these seem straightforward to fix, while others may require knowing the underlying business context.

How do you decide whether a problem is safe to automatically repair?

And are there situations where the data technically violates an expected rule but you would not change it because you don’t know what the correct value should be?
I’d especially appreciate examples from real projects.