how are you checking that your pipelines have complete sources?
Came across a problem recently that got me thinking about data quality checks.
One of our data sources looks healthy on the surface. requests are successful, ingestion is running fine, and there is nothing in the logs that indicates an issue. the problem only came up when I manually compared the output to the source and saw missing records and incomplete fields that should have been there.
The tricky part is that nothing is failing outright. the pipeline looks healthy unless you go and check the results wondering how others do this. what checks or monitoring do you use to make sure you're actually getting full data from a source and not just successful responses? Have you ever seen cases where a source returned partial data silently without obvious errors?