u/gp42

How are SRE teams handling the atomic ops stuff today? Restart pod, vacuum table, rotate creds, replay DLQ, force-delete a stuck namespace, drain a node.

There are tools for different pieces of this:

• Runtime / execution: Rundeck, Ansible Automation Platform, AWS SSM, Argo Workflows, Temporal...
• Shared / portable library: Ansible Galaxy is config not ops, StackStorm Exchange stalled, Rundeck has no job registry
• RBAC + per-action safety: AAP+SAML, custom homegrown, vault dynamic creds bolted on top
• Audit + traceability: whatever the runtime has, usually thin and tied to that runtime

Most teams I've worked with end up stitching pieces together. Something like AAP plus a private git of collections plus SAML plus a custom audit pipeline plus a Slack bot for triggers.

Questions I have:

1. What does your team's stack actually look like for this? Single tool? Stitched?
2. Can dev teams write their own playbooks, or does authoring stay gatekept by SRE/platform?
3. Is the setup actively painful (slow to iterate, hard to onboard, scary in incidents), or does it work fine once it's in place?

(Engineering org size context useful - 50 vs 500 vs 5000 changes the answers a lot.)

Trying to get a real picture of how teams handle this nowadays before building anything.

A few questions if you have a minute:

1. When you need to restart a pod, vacuum a DB table, clear a cache, rotate a password, what do you actually use? Script in a repo, Rundeck, Ansible, Jenkins, custom Slack bot, manual SSH?
2. Who's allowed to run it and how is that enforced?
3. Is there a shared library of these at your company, or does everyone write their own?
4. What's the worst part. Finding the right one, knowing it's safe, getting approvals, audit trail, something else?

I'm thinking about building an OSS thing in this space and don't want to assume too much. Would rather hear what's actually painful from people doing the work.

If you'd rather talk than type, 30 min call, no pitch. I'll write up what I learn across the conversations and send it back. Reply here or DM.

Inside AWS, the best practice is to not handle static credentials at all - your workload has an IAM role and the SDK signs every request with SigV4. The moment you call something outside AWS though (a SaaS API, a partner, another cloud), that's gone. SigV4 means nothing to a non-AWS service, so you're back to a long-lived API key sitting in Secrets Manager.

It turns out AWS already solved this - it can issue short-lived JSON Web Tokens (JWTs) for your workload's identity through AWS Security Token Service (via sts:GetWebIdentityToken). It's just not widely known, and there was no easy way to actually use it - or at least i did not find an easy way. So I built a proxy for it.

It's a small Go forward proxy. Point your HTTP client at it, and for each service you call it grabs a short-lived JWT from AWS STS, caches it, and renews it in the background - pretty much like a widely-known Sigv4 proxy. No app code changes. Anything that can validate an OIDC/JWT token can trust the call, with no shared secret. The token carries claims like account ID, org ID, region, and principal ARN, so the other side can do real authorization instead of just "valid key / invalid key".

Where it's useful: SaaS/third-party APIs that support OIDC, partner APIs that authorize you by your AWS identity, multi-cloud calls to GCP/Azure, on-prem services that trust AWS identity, and cross-account internal services.

Runs on EC2, ECS, EKS, Lambda. You need outbound identity federation enabled on the account and a role allowed to call sts:GetWebIdentityToken. Install via Docker (gp42/aws-outbound-jwt-proxy:latest), make build, or a release binary. Go, MIT.

Repo: https://github.com/gp42/aws-outbound-jwt-proxy

Curious if anyone here is already using outbound identity federation in prod - it's new enough that I haven't seen much discussion of it.