u/DavidPulaski

Disclosure for rule 2: I work at InvisiRisk, we build CI/CD security tooling. No links to us below. Flairing this as Education rather than Tools since it isn't about our product, happy to switch if the mods prefer.

On March 24 two malicious LiteLLM releases went up on PyPI, 1.82.7 and 1.82.8, live about 40 minutes before they were pulled. Part of the wider TeamPCP campaign that started with a leaked Trivy automation token. FBI FLASH on it, TLP: CLEAR so it's shareable: https://www.ic3.gov/CSA/2026/260702.pdf

The mechanism is the part worth knowing if you run a gateway. The package shipped a .pth file, and Python executes those at interpreter startup rather than on import. So it didn't matter whether your code ever called litellm. If it was installed and any Python process started, it ran.

It took environment variables, SSH keys, cloud credentials, Kubernetes service account tokens, and provider API keys.

That last one is why I think this is an MLOps problem specifically. LiteLLM sits in front of everything by design, so that one process has your OpenAI key, your Anthropic key, your Bedrock creds, whatever else you route through it. Probably the highest-value place in an ML stack to land a credential stealer, and for 40 minutes it was also the easiest.

So: has anyone actually gone back and confirmed either way?

Most of the obvious checks don't work here. If you pin loosely, something like litellm>=1.82, and a build ran in that window, you got it. Resolved manifests get discarded, so "what did we install on March 24" is often unanswerable months later. And a .pth payload runs before anything a scanner treats as import time.

One thing that does work and is faster than lock file archaeology. CloudSEK put up a public lookup for this incident: https://exposure.cloudsek.com/ai-supply-chain-incident

Worth being precise about it, since it answers a different question. Version history tells you whether you pulled the bad package. The lookup tells you whether your secrets turned up in what the attackers actually collected. It's closer to an outcome, and a 30 second check.

A hit still isn't proof of compromise. The FBI advisory makes the same point, that finding the dependency doesn't prove the code ran. Treat it as a reason to go dig, not as an incident on its own. And if your org does show up, keep it out of this thread.

Curious whether anyone confirmed, and how. Lock file history? Registry pull logs? Or did you just rotate everything and skip the reconstruction?

May Microsoft alert dropped 45 malicious npm packages targeting dev environments. 33 in the first wave, 12 more the next day. Dependency confusion. Same attack Alex Birsan pulled off back in 2021.

Still works. Still gets past everything.

Not because it's clever. Because of what your tools actually do.

SCA scanner sees a new package? It's comparing against a database of known bad stuff. A brand new public impostor isn't in that database yet, there's nothing to match against, gets flagged clean, installs without a second look.

Lockfile pinned to the right version? Fine, until someone adds a new dependency, or pins to latest, or a fresh install runs before the lockfile gets committed. That's when the resolver sees your private package name sitting on both a private and public registry, compares versions, picks the highest. Attacker published 9.9.9. Lockfile writes it down as legit on the next run like nothing happened.

Post-build scanning is just forensics at that point. Package already fetched. Install hooks already ran, with whatever access that grants inside the pipeline. You're not catching the attack, you're documenting it after the fact.

What actually stops it: Claim your internal package names on public registries first. Even empty placeholders work. An attacker can't register a name you already own.

Scope your internal packages, u/yourcompany, mapped to your private registry only. Get the config right and scoped names never resolve against public npm.

Watch your build-time network traffic. Internal package name pulling from a public source? Block it before the fetch happens. That's the actual moment this fails or succeeds.

That last one's the real fix honestly. Dependency confusion works because your build resolves names in the dark and trusts whatever comes back. Give it visibility into where each dependency is actually coming from and the whole attack falls apart.

Curious how everyone else is catching this one. Or is your team still trying to get the config perfect across every project and registry forever?

Install scripts aren't a vulnerability. They're infrastructure. npm and PyPI need them. Native modules get compiled, binaries get downloaded, config gets generated. You can't ban them without breaking half the ecosystem.

That's exactly why they're such effective attack surface.

In the last 18 months we've seen three major waves: Shai-Hulud (self-replicating, 25k+ repos compromised), Axios (March 2026, maintainer account hijacked, 100M weekly downloads), and s1ngularity (weaponized AI CLI tools for reconnaissance). All three executed at install time. All three harvested credentials, tokens, SSH keys. All three exfiltrated over the network.

And all three bypassed every standard control:

• SCA tools reason from manifests and known-CVE databases. A freshly published poisoned version isn't in those databases yet. The scanner doesn't observe what the install script actually does when it runs.
• Disabling install scripts (npm install --ignore-scripts) blocks the attack but breaks every legitimate build that compiles native code. Teams can't keep it on, so it's not durable.
• Secret scanners and artifact scans run after the fact. By the time they look, the script has already read the secret and made its network call. You're detecting the aftermath, not the exfiltration.

The gap is structural: none of those tools see the moment install-time code executes inside your build.

The only layer that observes that moment is the build pipeline itself. And that's where the defense model has to live, enforcing policy at the build boundary, not before or after.

The realistic take: install scripts aren't going away. The defense isn't to ban them. It's to treat npm install and pip install as untrusted code execution and build accordingly.

What does post-install script defense look like in your pipeline right now?

May Microsoft alert dropped 45 malicious npm packages targeting dev environments. 33 in the first wave, 12 more the next day. Dependency confusion. Same attack Alex Birsan pulled off back in 2021.

Still works. Still gets past everything.

Not because it's clever. Because of what your tools actually do.

SCA scanner sees a new package? It's comparing against a database of known bad stuff. A brand new public impostor isn't in that database yet, there's nothing to match against, gets flagged clean, installs without a second look.

Lockfile pinned to the right version? Fine, until someone adds a new dependency, or pins to latest, or a fresh install runs before the lockfile gets committed. That's when the resolver sees your private package name sitting on both a private and public registry, compares versions, picks the highest. Attacker published 9.9.9. Lockfile writes it down as legit on the next run like nothing happened.

Post-build scanning is just forensics at that point. Package already fetched. Install hooks already ran, with whatever access that grants inside the pipeline. You're not catching the attack, you're documenting it after the fact.

What actually stops it: Claim your internal package names on public registries first. Even empty placeholders work. An attacker can't register a name you already own.

Scope your internal packages, u/yourcompany, mapped to your private registry only. Get the config right and scoped names never resolve against public npm.

Watch your build-time network traffic. Internal package name pulling from a public source? Block it before the fetch happens. That's the actual moment this fails or succeeds.

That last one's the real fix honestly. Dependency confusion works because your build resolves names in the dark and trusts whatever comes back. Give it visibility into where each dependency is actually coming from and the whole attack falls apart.

Curious how everyone else is catching this one. Or is your team still trying to get the config perfect across every project and registry forever?

May Microsoft alert dropped 45 malicious npm packages targeting dev environments. 33 in the first wave, 12 more the next day. Dependency confusion. Same attack Alex Birsan pulled off back in 2021.

Still works. Still gets past everything.

Not because it's clever. Because of what your tools actually do.

SCA scanner sees a new package? It's comparing against a database of known bad stuff. A brand new public impostor isn't in that database yet, there's nothing to match against, gets flagged clean, installs without a second look.

Lockfile pinned to the right version? Fine, until someone adds a new dependency, or pins to latest, or a fresh install runs before the lockfile gets committed. That's when the resolver sees your private package name sitting on both a private and public registry, compares versions, picks the highest. Attacker published 9.9.9. Lockfile writes it down as legit on the next run like nothing happened.

Post-build scanning is just forensics at that point. Package already fetched. Install hooks already ran, with whatever access that grants inside the pipeline. You're not catching the attack, you're documenting it after the fact.

What actually stops it: Claim your internal package names on public registries first. Even empty placeholders work. An attacker can't register a name you already own.

Scope your internal packages, u/yourcompany/package, mapped to your private registry only. Get the config right and scoped names never resolve against public npm.

Watch your build-time network traffic. Internal package name pulling from a public source? Block it before the fetch happens. That's the actual moment this fails or succeeds.

That last one's the real fix honestly. Dependency confusion works because your build resolves names in the dark and trusts whatever comes back. Give it visibility into where each dependency is actually coming from and the whole attack falls apart.

Curious how everyone else is catching this one. Or is your team still trying to get the config perfect across every project and registry forever?

Been digging into supply chain attacks and noticed a pattern most DevSecOps teams aren't defending:

The problem: Package installation isn't always passive. npm lifecycle scripts and Python packages built from source can execute arbitrary code during install — before your app even imports the library.

Real examples:

• Shai-Hulud (Ruby gems)
• Axios maintainer compromise
• Nx attack last year

Why SCA/dependency scanners miss it: They look for known-bad packages in databases. But a freshly poisoned release hits your build before it's flagged as malicious. It runs before discovery flags it.

The gap: Most orgs have SAST, SCA, CNAPP, EDR. But nobody's really enforcing policy while the build runs. It's all pre-scan or post-detection.

How are you handling install-time execution in your pipelines?

Spent part of the weekend going through Socket's PolinRider writeup (the DPRK campaign hitting npm, Packagist, Go modules, and at least one Chrome extension, 108 packages so far per their count).

The package count isn't what got me. It's the delivery mechanism. The loader doesn't phone home to attacker infra at all. It hits public RPC endpoints on TRON, Aptos, and BNB Smart Chain, pulls down encrypted payload data, XOR-decrypts it, and evals. Your domain blocklist is useless because the destination is the same endpoint any legit web3 project talks to all day. Threat intel feeds have nothing to flag.

Also worth knowing if you're doing IR on this: they force-pushed rewritten Git history with backdated commits, so the malicious code looks like it's been sitting in an old legitimate commit for months. The repo page will lie to you. You need the activity log.

The RPC thing is what I keep coming back to though. The textbook answer is deny-by-default egress from build environments. In practice every team I've watched attempt that ended up buried in allowlist exceptions within a quarter and either loosened it until it was meaningless or turned it off. Feels like there's a gap between "correct on the whiteboard" and "survivable in production."

Anyone actually made default-deny egress stick in CI? What broke first?