Is there a missing pre-event layer in observability, or do current workflows already cover this?
Why is observability still mostly retrospective?
Most monitoring and observability workflows seem excellent at answering what crossed a threshold, what alert fired, and what happened after the incident became visible. But I keep wondering about the earlier window. In many systems, the alert is not the first thing that changes. Queueing, latency, cache behavior, load, memory pressure, or downstream coupling may start moving together before the visible incident.
So the question becomes: given a bounded historical trace, can we test whether the system entered a separable pre-event regime before the current alarm fired?
I’m not thinking of this as another alerting system. More like an offline audit of a past incident trace:
- start from one anonymized telemetry trace around an incident
- map raw metrics into a shared transition representation
- ask whether multiple channels began moving together before the current alarm
- compare that timing against the existing alarm or a tuned baseline
- classify the outcome as usable pre-event structure, no actionable signal, or unstable mapping
The distinction I care about is this: not “predict the future,” but audit a past incident and ask whether the telemetry had already entered a separable regime before the alert became operationally visible.
For people running production systems:
Does this sound like a real missing layer, or just overfitting the problem?
Do current observability workflows already cover this well enough?
Where would it fail in practice: noisy metrics, bad timestamps, lack of incident labels, false positives, trust, workflow integration?
I’m investigating this as part of a broader attempt to understand whether observability has a missing pre-event layer — or whether existing tools already cover it in practice and I’m just naming something teams already do informally.