Your Vercel preview is a per-PR QA environment that almost nobody opens. We built something that opens it.
Disclosure: I work on this, it is our product IronBee, link at the bottom.
Every PR on Vercel already gives you a complete running copy of the app, isolated, one per change. That is the QA environment teams have wanted forever, and almost nobody opens the URL. The reviewer reads the diff and merges.
So we built an agent that opens it and uses the product. It reads the change, works out what a user would do differently because of it, and does that. No scripts, no selectors to maintain.
Two parts that were harder than expected:
Not stopping at "it failed". We read OpenTelemetry from the app and ship an SDK for runtime introspection, so a finding is the failing request, the query that returned nothing, the exception logged at info, and the line in the diff.
Not trusting the agent. Ask an LLM to test a changeset and it will tell you it did. That is a claim, not evidence. We trace what actually executes during the run and check the changed lines mechanically. If part of the change never executed, we do not pass it.
It also runs in GitHub Actions for services that never get a URL.
Happy to answer anything technical. If it misses something obvious on a real PR, I would rather hear that.
Announcement post: https://medium.com/ironbee/your-ai-qa-engineer-now-runs-on-every-vercel-preview-16ca2d9dc03a