"A security tool builder challenged our AI reasoning council to a blind Terraform audit. Two rounds, no answer key. Here's exactly what the council caught."
Posted about a tool we built on r/aiagents last week. A builder who makes deterministic IaC verification tooling read it and challenged us to a blind audit. Two rounds. No planted issue list, no count, no answer key until findings were in.
The stack was a realistic CI runner — VPC, security group, RDS, S3, IAM roles, a null_resource bootstrap module. Terraform plan looked completely clean.
What was planted:
Three access-widening changes, each invisible to standard diff review.
Round 1 findings:
CIDR split on aws_security_group_rule.sshcidr_blocks = ["0.0.0.0/1", "128.0.0.0/1"]
No literal 0.0.0.0/0 anywhere. String search finds nothing. The two halves tile all of IPv4. The council did the coverage math and flagged it as open ingress.
Wildcard admin on aws_iam_role_policy.ci
Source showed only policy = var.ci_policy_json — a variable from the pipeline secret store. Resolved plan showed Action: "*", Resource: "*". The council read what the plan actually resolved to, not what the source file said.
AdministratorAccess via module.bootstrap.null_resource.attach
A shared remote module containing a local-exec provisioner. Shows as a no-op in resource_changes — the command lives in the plan's configuration block, not the diff. The council read the configuration block and flagged the exact command: aws iam attach-role-policy --role-name ci-runner --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
Three for three. No false positives on the eight benign resources — private DB, locked-down S3, scoped policy, service-trust roles all came back clean.
Round 2 — blind, no planted issue list:
Same approach. Same result. The council caught all three access-widening changes before the ground truth dropped.
The honest boundary:
The challenger made a point worth stating plainly for anyone considering where a reasoning layer fits in a real pipeline:
Reasoning finds and explains. A deterministic gate blocks and attests. Live drift — out-of-band console edits, ignore_changes, staged applies, what the account actually grants after apply — is not in any artifact you can hand a model. For that you need a tool querying live cloud state. That layer belongs to deterministic tooling.
Clean stack: reason to find → gate to block → verify reality.
How the tool works:
You describe your problem and paste your .tf files and plan.json directly. It generates a discovery prompt you run in your editor. Your editor returns a diagnostic report. You paste that back. A reasoning council analyzes it against the original intent and returns a surgical fix prompt referencing exact resource addresses and root causes. Your editor executes the fix.
Find. Instruct. Verify.
Free to use. No signup. No credit card.
👉 lookmood.me/ai-code-reasoner
The full exchange including both challenge rounds is in r/aiagents. The challenger's IaC verification tool is at github.com/amitpatole/verel — worth a look if you want the deterministic layer.