u/Swarm-Stack

i had a review chain set up in langgraph: three different models each pass over a plan before it ships, the idea being if one of them is wrong the other two catch it. worked fine until it didnt. a migration plan came through, all three reviewers approved it, and it dropped a column the nightly billing job still read from. none of them flagged it.

took me a while to see why. the three models werent really disagreeing, they were all reasoning from the same context i handed them, so they shared the same blind spot. adding a fourth model wouldnt have helped, it would just be a fourth read of the same framing. the miss wasnt "a model got it wrong", it was "nobody in the loop knew the billing job existed".

what actually fixed it was boring. the person who owns billing looked at the plan for ten seconds and said "that column, the nightly job reads it". not a smarter model, a different head with different context.

so i ended up building the thing i wanted out of that. you and your team plan in one live session, each holding a seat (your dba on schema, whoever owns billing on billing), and the models fill the seats nobody's in and double-check the calls the humans make. when nobody on the team actually knows the answer you pull in a verified outside expert who takes a seat too. the models are still there, just as gap-fillers and a second reader, not the whole review panel. what you get out is a versioned plan with the argument underneath, human and model both.

still rough, solo project. but the pattern im pretty convinced of now: model-only review chains converge because they share your framing, and the cheapest fix isnt another model in the same chair, its a seat held by someone whose context is different from yours.

curious if anyone here has gotten genuine disagreement out of a multi-model review chain without a human or a tool forcing different context in. every time ive tried, they just converge.

hit this on a project last year: the spec was technically correct, and when a requirement shifted six months later there was no record of what assumption the original decision rested on. the argument had happened in slack threads and in someone's head. what went into the spec was the conclusion.

wed re-litigate the original call, lose the original intent, and end up with choices that contradicted assumptions nobody had written down.

what i actually wanted wasnt a better spec format. i wanted the argument under the spec frozen in the same file. so i built SwarmStack.

the way it works: you and whoever needs to be in the room (your PM on product, your DBA on schema, whoever's relevant) plan in one live session. the AI fills the seats you dont have a person for and pushes back on the calls the humans make. what you get at the end is a versioned SwarmPlan -- the decisions, but also the contention that produced them. when requirements shift and you open it three months later, you can see what was argued, what got pushed back on, what assumption each call rested on.

swarm-stack.io. still pretty rough. genuinely curious whether the "record of the argument" framing resonates here, or if most people feel the spec-as-conclusion problem differently.

we had three models reviewing the same architecture plan and they kept agreeing on the wrong thing. first theory was training similarity, same RLHF fingerprint, same convergence. that wasn't it.

the actual problem was the prompt. asking "what's wrong with this plan" from the same starting point means each model is flagging what its weights already identified as weak. they were doing that fine. but "find problems from wherever you're looking" and "find problems looking from one specific angle" are different instructions.

switched each reviewer to a specific mandate: QA had to return break scenarios only, formatted as scenario plus conditions that trigger plus estimated blast radius. infra had to return load and retry failure points only. product had to return user-visible impact only. different output schemas, not just different prompts. each reviewer had to go deeper on one class instead of skimming all of them.

overlap between branches dropped about 35%. more importantly, the infra branch surfaced a retry-cascade scenario the general review had tagged as low risk. it wasn't low risk, it just looked that way from a general-review frame because the blast radius only shows up when you're specifically looking at load thresholds.

still haven't solved: findings that sit at the boundary of two failure classes. QA and infra both surface the same root cause but frame it differently. deduplicate by class first and you keep both framings, which is probably right since they carry different blast radius estimates, but it inflates the report. anyone found a clean way to handle cross-class findings without collapsing them or drowning in them?

we shipped an auth feature that passed spec review three times. product signed off, backend signed off, i signed off. four weeks later someone filed a bug: two users could hit the confirmation endpoint simultaneously with the same token and both get through. obvious race condition. the spec never mentioned concurrent access because none of us had been assigned to think about it.

the spec wasn't wrong about what it described. it just never described the failure surface. everyone in the review was asking 'does this design implement what we want?' nobody had a mandate to ask 'what breaks this under real conditions?'

what fixed it was stopping collective approval and starting role mandates. backend: state consistency under concurrent load. product: user-visible blast radius if it fails. security: trust boundary assumptions and edge inputs. reviews got shorter and more specific — 15 minutes per role looking for something concrete beats an hour of general agreement.

built this pattern into swarm-stack.io — each seat in a planning session carries a specific failure-class mandate. curious whether others here have formalized this at the spec-review stage or whether the attack angle usually just depends on who's in the room.

we had a plan-review step in a langchain workflow. kept getting confident approvals on designs that broke later.

first attempt to fix it: route the plan through three different models. gpt-4o, claude, gemini. figured they'd catch different things. they didn't, really. they disagreed on wording sometimes. on substance they converged 80% of the time to whatever framing the original plan used.

what actually worked: role isolation. instead of "review this plan," each chain gets a specific mandate. "you are QA. find the scenarios that break this." "you are backend. find what doesn't scale." "you are product. find what users will notice if it goes wrong." each one is explicitly looking for its failure class, not trying to be comprehensive.

the disagreement that came out of that was useful. QA found the offline case. backend found the retry budget assumption. neither was catching the other's failure class, which meant both got caught before shipping.

the failure mode with multi-model routing is that you're still asking everyone the same question. model diversity matters less than question diversity. an agent mandated to find failure class X finds different problems than an agent mandated to be a balanced reviewer.

curious whether others have moved away from multi-model toward role-isolated mandates, or whether the variance source in your setups is something else entirely.

last week a standup caught something that would have died in planning. we were five minutes in when someone asked "wait, what happens when the device is offline" and the answer was basically "the spec doesn't say."

the offline case wasn't a weird edge. it was obvious in retrospect. the problem was that one person wrote the spec and one person reviewed it, and both were reasoning from the same mental model. nobody pushed back. the reviewer confirmed what the author already believed.

we'd built the whole thing correctly against a spec that had a hole in it.

so i changed the process. instead of one person writing and another rubber-stamping, i now set up role-based personas before the spec freezes. product, QA, backend. each one argues it from their own angle. the QA persona asks the offline question. the backend persona asks about retry budgets. the product persona asks whether users will even notice the degraded state. they're not agreeing with each other, theyre trying to poke holes. the output of the argument is the spec, with the contention sitting underneath it so you can see why the final call was made.

the thing i've been building around this is at swarm-stack.io if you want to look at it.

curious whether others have hit the same failure mode. single-author spec with a single reviewer is effectively no review at all. or is that too strong? is this a people problem that process cant fix?