u/PackPretty3479

The fix took 10 minutes. Finding who owned the asset took 4 days and 6 escalation emails.

I need to vent about something that happened last week cause I'm still annoyed.

We had a straightforward misconfig on a server. Open port that shouldnt be open. Yeah, basic stuff. The fix itself was maybe ten minutes. Log in, update the config, restart the service, call it a fix.

The ten minute fix took four days.

Day one: I find the finding in our scanner, but no owner is listed. I check the CMDB, the owner field says IT Operations which in our org means exactly nobody. I send an email to the distro, got a big ugly nothing.

Day two: I dig through old tickets and find the server was originally set up by an engineer who left eight months ago. His replacement doesnt know anything about it. I escalate t the infra team lead.

Day three: Infra says its not theirs, it was built for an app team project. App team says they never owned it, infra stood it up for them. I escalate to both directors.

Day four: A director finally claims it, assigns an engineer, the fix happens in ten minutes just as it should have happened 4 long days ago. Ticket closed.

So this small thing tool 4 days, 6 angry emails, 2 director escalations. And this isnt even unusual. This is maybe 40 percent of our findings. The technical fix is never the bottleneck. The bottleneck is figuring out who the hell is supposed to do it.

How do you handle ownership at scale when CMDB fields are stale within a quarter?

reddit.com
u/PackPretty3479 — 2 days ago

We wanted to automate remediation. Got blocked because nobody trusted the asset data enough to let automation touch it

Spent the last quarter building out a remediation automation pipeline. The plan was simple enough, when a CVE hits a certain threshold, the system checks if the asset is production and internet facing, identifies the owner, creates a ticket, and if the risk is high enough, triggers the patch deployment directly through our endpoint management tool. Intended a closed loop, no human handoff.

Got the whole thing wired up, tested it in dev and it worked beautifully.

Then we tried to point it at production and everything fell apart. The automation didnt really break, the thing is nobody would sign off on letting it run against real assets. The infra team wanted proof that the asset list was accurate before patches star ted deploying. The security team wanted confirmation that the ownership mappings were current and not outdated stuff from previous quarters audits. The app owners wanted to know how we'd verify a patch didn't break their service. Every objection came back to the same root which is nobody trusted the underlying asset data enough to let automation act on it without a human in the loop.

So now we have this beautifully engineered automation pipeline sitting idle while we go back and manually validate the asset inventory it depends on. The irony is killing me,, we built automation to save time, but we cant use it until we spend months verifying the data it needs, which is exactly the manual work the automation was supposed to eliminate.

If anyone has ever been in this gap, I would really love to hear how they handled it. Even if you havent, I need ideas on how you can get from having asset data to trusting the asset data enough to automate against it

reddit.com
u/PackPretty3479 — 8 days ago