u/Jaded-Share-6329

Hi, I'm a bit annoyed about the fact that I can lose logs, and I'm starting to think maybe I should fix it generically instead of tailoring special alerts and rules in the observability layer.

• Someone pushed a change and it broke the log format, so now there's a hole in the logs — but everything's green. The new format causes a parsing error into the index, and a parsing-error log often goes unnoticed. The solution I see is an alert with a threshold, but then someone still has to go check what's actually going on.
• Sometimes the ELK server takes the logs with a lag, or even crashes silently, and you end up with a hole in some indexes.

How do you guys catch this? Logs/events silently stopping, with no error to fire on. Do you alert on expected volume, or is there something less hacky? Curious if everyone just eats it, or if there's an actual approach.