u/Lance_Saul_85

I do not know how other teams put up with this. Our security scan runs on every PR and re-reads the whole repo each time, it is always the slowest step by a mile, slower than the build and the tests put together. During an outage last month it sat on a one line fix long enough that a lead just force merged past it and I did not blame him.

And now that is normal. People skip the scan whenever they are in a rush, which is most of the time then it still shows green on the dashboard so on paper nothing is wrong. A gate everyone routes around is not really a gate, it just looks like one in the audit.

So I am trying to work out how you make it fast enough that skipping it stops being the obvious move. Incremental scans that only look at what changed. Something in the editor you see it while you type instead of waiting on the pipeline. Both maybe. What is working for you, and did you give up coverage to get the speed.

I think we have been staring at the wrong number when it comes to measuring container security. Total CVE count is what everyone tracks. It is a snapshot, a stock. It tells you what a scanner found today and nothing about tomorrow. Two images with the same count can behave completely differently over 6 months one stays quiet, the other throws a new critical every week.

What we started tracking is frequency of how much time passes on average between new CVEs appearing in a given image. A base dragging in a huge pile of packages gets a new vuln constantly because there is so much surface for a disclosure to land on. A minimal one goes quiet for weeks because there is just less in it that can be found vulnerable.

And that resulted into far better prediction on triage load. The noisy images had the biggest number on day one and they were the ones with the most surface accumulating new findings over time.

Have you tracked frequency like this or changed how you measure container risk?

Spent a while assuming spend was climbing because usage was climbing. It wasnt, request volume was flat, cost per request was the thing that moved.

Two things once we instrumented it properly. There was a retry on timeout that in certain failure modes fired three times on one request and every one of those was a full price call. Separately, our context had grown because people kept appending to the system prompt over months and nothing ever removed, so it was up near 4k tokens of accumulated instructions with some of it contradicting other parts of it.

The retry was a bug and thats fixed. The prompt is more of a people problem, everyone who added a line had a reason at the time and nobody wants to be the one who deletes someone else's guardrail.

Anyone got a sane process for stopping a shared system prompt turning into that. We've talked about ownership and reviews but it feels like it needs to be more boring and automatic than a review.

Been chewing on this for a while. We run a SAST tool on our Go services and I have realized none of us can say how accurate it is. It throws findings, we work through some, we ignore a lot and we have never once measured how many were real versus noise. We just vibe it.

What set me off was two scanners disagreeing hard on the same repo. One flagged 30 criticals while the other found 6. They cannot both be right and I have no way to tell which one is lying to me. Everyone sells accuracy now, especially with all the AI code we are pushing but no one hands you a way to check the claim.

How are you all measuring whether your scanner is right, not just whether it is loud.

Postgres, Redis, nginx. We moved everything onto continuously rebuilt minimal images about 6 weeks ago to cut down the Debian CVE noise we were drowning in and it helped.

Every bump still goes through the test suite. No shell makes debugging painful though.

Anyone else do this and still feel like it was worth it?