How to automate weekly manual QA smoke tests using local LLM browser agents?
Hi everyone,
I’m looking for some advice on automating a tedious weekly task at my company, and I want to see if anyone here has successfully tackled a similar problem using modern AI tools.
The Situation:
Every week, our team has to manually click through a dozen or so test scenarios on our application across various client environments. It’s basically repetitive smoke testing that takes up too much time.
What I want to achieve:
Write the test scenarios in plain natural language as a simple list of steps (e.g., "Log in, go to settings, change the username, verify the success message").
Feed these steps into an AI Agent that can interact with the browser, analyze the page in real-time, and execute the steps.
Crucial requirement: The tool must not just generate Playwright/Selenium code. I need an agent that actively navigates the page and can adapt on the fly if the layout or CSS selectors change slightly, as long as the underlying logic remains the same.
Privacy is a must: Because we deal with sensitive client data, I need to power this agent using a local, self-hosted LLM (running via Ollama, vLLM, etc.) so no data leaves our infrastructure.
I’ve heard of frameworks like Browser-Use, LaVague, or Skyvern, but I’m not sure how well they handle local open-source models (like Llama 3 or Mistral) for reliable E2E testing, or if there are better enterprise-ready/open-source alternatives out there.
Has anyone built a similar pipeline? What tools, frameworks, or specific local models worked best for you to achieve high reliability without constant maintenance?
Thanks in advance for your help!