u/Primary_Cupcake8557

AWS has real frameworks for this beyond checking boxes: Well-Architected, Resilience Hub, and RA2 assessments cover 8 dimensions including redundancy, durability, change management, and testing.

Most of what we have internally is ad hoc. No one's ever run a formal review, we just sort of assume things are resilient because nothing's broken yet.

I'm trying to build a case internally for actually doing one of these assessments properly, probably starting with Resilience Hub since it's native and free to try.

For non-AWS or multi-cloud setups, how are you actually scoring this? Continuous posture tool or point-in-time assessment, and how often do you re-run it

Had a scare recently, not a full incident thankfully, but close enough to be a wake-up call. Realized our "backups" only cover data, not the IAM policies, networking, and dependencies that actually make the app run.
We went digging afterward and found out Gartner just created a whole category for exactly this gap, called CAIRS, because traditional DR assumes you're restoring data, not rebuilding infra from scratch.
The idea of treating infrastructure as reconstructible code rather than something you restore from a snapshot makes a lot of sense once you think about it, but I have no idea how mature the tooling actually is in practice.
Has anyone tested a full environment rebuild from IaC after a genuinely destructive event? How long did it realistically take end to end, and what broke that you didn't expect

Used to think "resilient" just meant Multi-AZ RDS plus a backup schedule. Turns out it means the whole environment (VPCs, IAM, DNS, load balancers) can survive a failure and rebuild within your RTO/RPO, not just individual pieces staying up

Multi-cloud makes this way harder to validate continuously. How is everyone actually testing this beyond chaos engineering on one service at a time?

I have been comparing Spacelift alternatives and noticed that many are not direct replacements. They solve different parts of the infrastructure lifecycle.

The main options I found are:

- Terraform Cloud for managed Terraform state and runs

- Firefly Ai for cloud inventory, IaC codification, drift remediation and infrastructure recovery

- env0 for multi-IaC automation and cost controls

- Scalr for Terraform governance and multi-tenant management

- Atlantis for self-hosted Terraform and OpenTofu pull-request automation

Choosing an alternative seems to depend on what's missing from the current workflow. Some teams need better orchestration and governance, while others need visibility into unmanaged cloud resources, developer self-service, drift remediation or infrastructure recovery.

For those who have evaluated or migrated away from Spacelift, which option did you choose?

We went heavy on Azure about two years ago and built what looked like a strong resilience story: availability zones, paired regions, Front Door and Traffic Manager, Backup and Site Recovery, plus some experiments with Chaos Studio. The first real regional wobble reminded us that slides are one thing, operations are another.

Just picking a paired region was not enough when identity and networking were not thought through, and monitoring was not included. Backup and Site Recovery were only comforting once we had run actual failover tests against real RTO and RPO targets, not just health checks. Chaos testing was useful, but only for teams willing to design non-trivial scenarios. Governance turned out to be the hardest bit: keeping landing zone baselines in sync across many subscriptions and tenants via Azure Policy, especially with hybrid in the mix. Terraform or Bicep helped, but drift remained a problem.

What has made this more manageable for us is having cross-subscription and cross-cloud governance visibility into assets and IaC: being able to see which Azure resources are driven by code and where live state has drifted from definitions. Combined with the failover testing, that is what turned "paired regions plus Site Recovery" from a diagram into something we actually trust. For those of you running production on Azure, GCP or any cloud, which resilience patterns have worked for you, and how are you keeping them from decaying over time?

i used to think good monitoring meant collecting everything Prometheus could scrape and building big Grafana walls. After a few ugly incidents as the on call, i care a lot less about pretty charts and a lot more about fast signal.

Our monitoring only got useful when we tied it to service level objectives and error budgets, not "all metrics forever." We moved to a handful of user journey synthetics, a small set of service health metrics, and logs that were structured enough to query under stress. The infra views came after that. One surprise was how often monitoring itself drifted: alerts disabled temporarily, thresholds changed without review, dashboards referencing resources that no longer existed. We ended up treating alerting rules and dashboards as code too, which is the only reason drift aware tooling catches monitoring rot instead of just cloud resource drift and it has been key to keeping monitoring aligned with reality.

If you have been on the hook for fixing things at 3 am, how did you decide what to monitor and keep it from rotting as fast as the systems it covers?