Are we missing a new QA discipline for AI systems?
I’ve spent years working with traditional QA and test automation, where the mental model was relatively straightforward:
requirements → input → deterministic behavior → expected result → assertion.
AI systems are making me question that entire model.
With an LLM, RAG system or agent, the same input may produce multiple valid outputs. A system can pass functional tests and still hallucinate. An agent can successfully complete a task but use the wrong tool along the way. A RAG pipeline can return a plausible answer grounded in the wrong document.
And suddenly:
>!expected == actual!<
doesn’t describe quality particularly well anymore.
The more I study this, the more I think experienced QA engineers may actually have a very useful foundation for AI systems.
Not because we know machine learning.
Because we’re trained to distrust assumptions.
Someone says:
“It works.”
QA asks:
Under what conditions?
How many times did you run it?
What happens at the boundary?
What happens with bad input?
Can I reproduce the failure?
What happens when one dependency changes?
What evidence do we actually have that this is safe to release?
I’m starting to see an emerging intersection of:
traditional Quality Engineering
LLM evals
reliability engineering
RAG and agent testing
adversarial testing
AI red teaming
production observability
I’m not suggesting QA should suddenly own ML research, security, safety and MLOps.
Quite the opposite.
I’m trying to understand where the boundaries should be.
For people here already testing AI products:
What parts of AI quality genuinely belong to QA?
And if an experienced automation engineer wanted to move into AI without throwing away years of QA experience, what would you consider the essential new skill set?