u/Own_Drink3843

Almost lost a chunk of a client's account to a bad Terraform apply last week. Someone ran the wrong workspace against the wrong environment and it took down a good chunk of a staging-adjacent VPC before anyone caught it.

Made me realize our backup strategy covers data stores but nothing else, IAM roles, networking, cross-service dependencies were all missing from the recovery plan. We got lucky it wasn't prod.

This is apparently the exact gap the new CAIRS category is meant to close: infra as rebuildable code, not just restorable data. Reading about it made me realize how much we've been conflating "we have snapshots" with "we can actually recover.

How often is everyone actually testing full-account recovery for clients, and does anyone have a realistic cadence that doesn't eat a ton of engineering time?

Compliance is asking for proof we can recover cloud infra within a defined timeframe. "We have backups" apparently isn't going to cut it.
What actually satisfies an auditor here, documented RTO/RPO per workload, a tested rebuild log, something else? Anyone been through a DORA or NIS2 audit recently who can share what held up

I am doing DevSecOps in a place where almost everything is defined as code and goes through CI or CD. We have tried plenty of DevOps security lists and frameworks and most of them felt great in slides but rough in real pipelines. The stuff that stuck was boring but effective: modules and Helm charts with secure defaults baked in instead of one off hardening after the fact. Secrets living in an actual secret store rather than whatever environment variable seemed convenient that week. Static analysis, dependency scans and IaC checks wired into pull requests so they hit developers early instead of as a surprise in staging. And then, once infra was mostly in code, a reality check: drift detection and cloud level scanning to find unmanaged resources and sneaky security group or IAM changes, pushing fixes back through Git before security reviews turned into archaeology. If you are in a similar position, which practices have survived real delivery pressure and which advice sounded good but made everything slower and more fragile?

Coming at this from an SRE and cloud security angle, we went through two provider side incidents in the last eighteen months that forced us to rethink what resilience meant here. Uptime graphs and status pages were nice, but they did not help much when a managed database service had issues or our control plane was flaky. The tools that mattered were the dull ones we had half implemented: native backup and disaster recovery that we had bothered to test, a cross cloud disaster recovery platform we bought for compliance, and governance that enforced tagging and backup rules everywhere. The big unlock was having infra in IaC plus something watching for drift and unmanaged assets. Being able to see which resources were codified, where live state disagreed with IaC, and which changes were pure clickops made decisions during the outage much less hand wavy. If you have had to prove this to risk committees or live through provider problems, what tools or patterns saved you. Are you leaning on native cloud features or have separate resilience or governance or IaC drift platforms actually earned their keep?

I have been in an AWS only shop for years now. We have done the Well Architected reviews, drawn the multi availability zone and multi region pictures and talked about pilot light versus warm standby until the whiteboards wore out. Some of that held up, some did not. Resilience on AWS is supposed to be straightforward: spread across availability zones, add a second region, lean on managed services, drive everything through IaC, test with chaos.

What I have seen is just this one workload in a single availability zone, backups tied to cross account IAM roles that quietly lost permissions, and runbooks nobody remembered under pressure. The thing that helped was shrinking the pattern set and getting much better visibility: one place that shows which resources across accounts and regions are managed by IaC, where drift has happened, and how to push fixes back through code. For ppl running serious workloads: have you found IaC plus drift aware tooling useful for keeping active, warm standby or pilot light setups in sync with reality or does it end up as another dashboard nobody trusts under pressure?

We are running about 95 AWS accounts and somewhere around 600 Terraform modules across three teams. The drift problem has gotten to a point where I am embarrassed to say how many tickets and platform team attention goes to it. The issue is not that we do not know drift exists. We know each one is small, but six months later, the plan output is so noisy with expected drift that people stop reading it, which means real unintended changes get buried. We had an actual misconfiguration slip through last quarter because the engineer skimming the plan categorized it as more drift garbage. That one cost us a few hours of incident recovery we did not budget for. We have tried scheduled drift scans with Terraform plan run in CI on a cron job and routing alerts to Slack. The alerts get ignored within two weeks because there's too much volume and no clear ownership.  What tooling or process changes have moved the needle on this for teams running IaC with this kind of account sprawl? Not looking for the obvious write better IaC from the start answer. Looking for what's working operationally for people who are already mid-mess. 

I have tried everything I can think of and the pattern keeps repeating. We built out what I thought was a solid internal platform. service catalog, pre approved modules, guardrails baked into the CI pipelines. Devs are supposed to provision through the catalog, everything gets tracked in state, auditable, the whole thing. and it works great for about 80% of provisioning. The other 20% happens when someone is blocked, under pressure, or just does not know the catalog has what they need. They go to the console or use their own ad hoc terraform that never gets merged back. Suddenly there is an RDS instance or an ECS task definition sitting outside of anything we control. The troubling part is not that it happens once. It is that it compounds. you find it six weeks later during a cost review or an incident and by then it is load bearing. No one wants to touch it. It just stays there, unmanaged forever. I have thought about harder restrictions on IAM permissions but that creates a support ticket flood every time someone has a legitimate edge case. Automated discovery helps surface it after the fact but does not stop it happening. drift detection tools catch it but the signal gets lost in the noise when you are running more than a handful of accounts. If you have solved this, what's working? I am specifically interested in how people are closing the gap between the what our platform provisions and what exists piece, without needing humans to reconcile. Bonus points if whatever you are using helps when you need to recover or rebuild an environment, not just audit it.

Running a 30 engineer org with 12 AWS accounts and 20 Azure subscriptions. Currently on HCP Terraform and hitting the wall on the things it doe not do well, plus too high renewal quote. State management and remote runs are fine.

What is not fine: no visibility into resources that were provisioned outside Terraform, drift detection that only covers registered workspace, and policy enforcement that requires a lot of Sentinel work to get meaningful. Looked at env0 as a potential move. The workflow customization looks better and the cost management features are interesting but from what I can tell it has the same blind spot as HCP Terraform when it comes to cloud resources that exist outside managed workspaces. If your IaC coverage is incomplete going in, neither way helps you close that gap. What I want is a platform where the IaC orchestration and the cloud asset inventory are the same product, not bolt-ons, at a reasonable price. Is there anyone who found something that treats unmanaged cloud resources as a first class problem rather than an afterthought?

First attempt we locked down IAM hard and forced everything through a service catalog. Developers hated it. Unsurprisingly, ticket volume to the platform team tripled and people found workarounds. Those workarounds became load bearing infrastructure within three months and now we had shadow IT outside IaC coverage AND a team that resented the platform. Somehow ended up worse off than before. Second attempt at a different org we loosened the guardrails and focused on developer self service cloud provisioning with better experience. Got higher catalog adoption but the fundamental problem didn't go away. Someone provisions directly during an incident because the catalog path is too slow and suddenly the unmanaged resources accumulate again. They don't show up in state and when you go to calculate your live cloud footprint for cost, compliance, disaster recovery, the number is always higher than what your IaC says.

The part that gets me is this that  gap between what your IaC state says and what is running is a structural problem. The tooling doesn't close it by itself so humans are supposed to close it manually and they don't because there are always higher priorities. Is there something that handles continuous discovery and IaC generation for resources outside your defined provisioning paths. Not a one time import . Ongoing reconciliation between live cloud footprint and IaC state at scale. Curious if anyone has solved this or if we are all just living with the gap. 

Important: not looking to replace orchestration with more orchestration.

We've been on Spacelift for a while. The workflow automation is solid and the runner infrastructure works well for us. The gaps we keep running into are on the visibility side. Spacelift orchestrates what we tell it to orchestrate but has no awareness of resources that exist outside its workflows. We have a meaningful chunk of infrastructure that was never brought under IaC and Spacelift doesn't help you discover or manage that. Drift detection only covers stacks it knows about, which is not the same as your actual cloud footprint. What we need is something that continuously scans across cloud accounts, surfaces resources outside IaC coverage, and ties that visibility back into the IaC workflow rather than treating it as a separate concern. 

Has anyone made this switch and found a Spacelift alternative that handles both the orchestration and the cloud asset visibility side? Specifically interested in whether the migration was painful and what the net improvement looked like in practice.

Edit: Appreciate the detailed replies. The biggest thing I underestimated going into these evaluations was how many platforms assume IaC coverage is already complete. Feels like the actual problem for us is still visibility into resources outside managed stacks. Firefly ai has been interesting on that side so far because it starts from what exists in the accounts.