u/Chris__Codes

I’ve been checking how to build a disaster recovery plan for cloud dependencies and how to automate as much remediation as possible. The plan is to have the system auto-restart services or rollback deployments when things crash. 

But when I read up on AIOps, I saw recommendations to make sure a human is involved for anything high-risk. It makes sense because you do not want an automated script taking down a production database by mistake. But now I am trying to figure out where to draw the line exactly between guided recommendations and full automation. 

If anyone here relies on machine learning for root cause analysis, which actions do you let the system execute on its own, and what requires manual approval?

I was looking into how to build database architecture for an app and trying to decide between self-hosting Postgres or paying for a managed service. Most call their database managed, but vendors may have completely different definitions for the word “managed”.

Some providers just patch the OS and leave all the failover, capacity planning, and scaling decisions to engineers. If there is a traffic spike and a team has to manually resize the compute, then it is not actually scaling itself. I’ve started looking at serverless options that dynamically scale to zero when idle and handle the patching, backups, and intra-region failover automatically.

If you’re running managed databases in production, what tasks did you expect the provider to handle that you ended up doing yourself?

I read a guide about non-human identity governance and there's one point I keep thinking about. Most advice says every machine identity needs an owner. That could be fine in principle but the ownership I've seen recorded in practice is a team name, a repo, or whoever's name was on the ticket that created it sometime ago.

None of those tell you who can rotate the thing safely, who knows what depends on it, or who has the authority to kill it while an incident is running. Which makes me think ownership on a spreadsheet and ownership that functions under pressure are two different things, and only one of them gets recorded.

I want to know how this works in practice for people running real infrastructure. Is ownership assigned to a person or a team in your setup, and does that survive the person leaving? And when you need to revoke something urgently, do you actually know where all its dependencies are, or do you pull it and wait for something to start alerting?

When I first looked into wiring external tools to an LLM through an MCP server, I'd simply put the API keys in environment variables and let the agent call whatever it needed. That's what I'd normally do for a normal backend service.

I read around it, and what I had not thought about is that a backend service has a fixed code path. You know which line makes which call, so you know what the key gets used for. An agent picks its own tools based on a prompt, so the same key now sits behind whatever it decides to do.

Your vault still tells you where the key lives and when it rotates. It just doesn't tell you who authorized a particular call or which tool it ended up reaching.

The things I keep seeing suggested are scoping credentials per server rather than one key for everything, and injecting them at runtime instead of leaving them in env vars.

For the people who are running multi-tool agent setups, how are you handling backend access do you custom guardrails around what the agent can reach, or do you treat it as a normal service and accept it?

A while ago on a post I made someone pointed out that if a workload uses an identity to reach a secret instead of holding the raw value, most of the rotation worry goes away. Another person mentioned that moving their CI to OIDC and short-lived tokens got rid of most of their anxiety about old keys sitting in git history. Been reading around it since.

I keep seeing that federation only covers the part of the stack that can participate. A pod authenticating to S3 or a runner minting a token for AWS is fine, but Stripe doesn't do OIDC, and neither does an on-prem SQL Server or a webhook secret someone could’ve emailed over. Those exist as strings regardless, so there's still a lifecycle to run for them no matter how much of the cloud side you federate.

What I can't work out is the proportion. Whether the leftovers end up being a handful of third-party keys you could almost manage by hand, or whether it stays substantial enough that you're running both properly.

For anyone who’s moved to workload identity, roughly how much of your credential inventory went away, and what stayed? Also did the leftovers turn out to be the annoying ones, or were they easy to corral once the cloud keys were gone?

I’ve been fiddling with some agentic workflows and I have come to notice an issue with how agents handle web scraping.

Normally, when you hand an agent a URL, it scrapes the page, and it drops the entire markdown payload into the context window. If the URL was just a login wall, a generic navigation page, or completely off-topic, you still burn the tokens to figure that out.

I was checking out a tool that approaches this by returning page metadata (like the page_structure, category, and ranked snippets) alongside the text. The agent can evaluate the structure and category to decide if the page is actually useful before it processes the full payload.

How do you handle this in your projects? Do you have a pre-processing step to catch login walls and nav pages, or are you just passing the raw markdown straight to the model?

I always assumed deleting a key from a file was enough but from what I’ve learned that’s not how git works

Pre-commit hooks and CI only look at the diff. So if a key gets committed and you delete it in the next commit, every check turns out OK from then on, but the value is still sitting in history and still valid. The recommendation is to scan full history on a schedule and treat anything you find as exposed, and rotate it, even though it's long gone from the current files.

So the scan is really just telling you a leak already happened, and rotation is the part that actually contains it.

I have side projects from years ago where I don't remember what was committed before I knew better. Some of those keys are probably still valid.

Do you run scheduled history scans, or just pre-commit and CI? And when something surfaces from years back, do you rotate it or make a call based on whether the repo was ever public?

Most of my day-to-day is spent working with React and TypeScript, but I have been picking up Python recently to get into data pipelines. I read a tutorial about building a price monitoring agent, and it covered some failure points that happen when targeting heavily protected sites like Amazon or Walmart.

Two specific quiet failures that will break these pipelines in production without immediately throwing obvious errors were pointed out:

1. Retrieval failure, where your script sends a request and does not crash, so you think it worked, but in reality, the target site served a blank JavaScript shell, an error 1020, or a Cloudflare block instead of the actual HTML product page.
2. Extraction failure where you successfully get the data to the LLM, and it finds the price. But across different runs, the LLM changes the data type, returning a float one time and a string in the next run. And this data drift can quietly break your downstream database writes.

For those of you writing Python scraping scripts in production, how are you handling that extraction data validation? Do you rely heavily on structural schema parsing libraries to keep your LLM outputs strictly typed, or do you have a different setup?