u/MoneyMediocre4791

I’ve been thinking about what happens when AI starts writing a significant portion of our test suites.

Generating tests is becoming easy.

Knowing whether those tests are actually adding new coverage feels like the harder problem.

I built a small tool that embeds Playwright tests into a semantic vector space, clusters similar scenarios, and visualizes them as a graph.

A few things it surfaced on one of my projects:

* Multiple tests covering essentially the same flow with slightly different wording.

* Tests in completely different files that were near-identical in intent.

* Dense clusters around heavily tested functionality, with sparse regions highlighting areas that had very little coverage.

Here is the tool - 100% free, open source, run locally with your own LLM api key (currently supports openai and anthropic):

Simply run: `npx "@testchimp/semantic-graph" visualize --tests-dir ./tests` (remove the "" - not sure why the @ turns in to a u/ without it)

Set your llm api key as per: https://github.com/TestChimp/semantic-graph#environment-variables

tests semantic-graph

Been seeing this question come up with teams that attempt to retrofit their workflows for agents.

“Why keep test plans / stories / product context in code? Just expose Jira through MCP tools.”

Something like:

• list_stories
• get_story
• update_story

Voila! Technically the agent now has access to everything.

But access ≠ understanding.

The difference is similar to someone who has "read the entire library" vs someone "with a library card".

A library card technically gives access to every book. But someone who has actually read the library understands relationships, patterns, structure, context, etc.

Apply the same logic to your code. Imagine your codebase was stored as individual files, in a remote SaaS, and accessed purely via MCP tools:

• list_files
• read_file
• upsert_file

Technically your agent has the entire codebase available. But practically, losing out a bunch of capabilities:

• local indexing optimized for retrieval
• folder structure as implicit context
• grep/find across everything
• reading nearby context naturally
• faster iteration during multi-step reasoning for chain of thought

The agent doesn’t just access the code - it starts understanding the shape of it.

The same principle apply to product knowledge too. If stories, tests, and knowledge lived in a native/code-like form, agents can build a richer model of the business instead of pulling one record at a time through tools.

Curious if others have thought about this.

Do people think MCP + tools is sufficient? Or is there something fundamentally different about agents having native/local access to structured context?

Been seeing this question come up with teams that attempt to retrofit their workflows for agents.

“Why keep test plans / stories / product context in code? Just expose Jira through MCP tools.”

Something like:

• list_stories
• get_story
• update_story

Voila! Technically the agent now has access to everything.

But access ≠ understanding.

The difference is similar to someone who has "read the entire library" vs someone "with a library card".

A library card technically gives access to every book. But someone who has actually read the library understands relationships, patterns, structure, context, etc.

Apply the same logic to your code. Imagine your codebase was stored as individual files, in a remote SaaS, and accessed purely via MCP tools:

• list_files
• read_file
• upsert_file

Technically your agent has the entire codebase available.

But practically, losing out a bunch of capabilities:

• local indexing optimized for retrieval
• folder structure as implicit context
• grep/find across everything
• reading nearby context naturally
• faster iteration during multi-step reasoning for chain of thought

The agent doesn’t just access the code - it starts understanding the shape of it.

The same principle apply to product knowledge too. If stories, tests, and knowledge lived in a native/code-like form, agents can build a richer model of the business instead of pulling one record at a time through tools.

Curious if others have thought about this.

Do people think MCP + tools is sufficient? Or is there something fundamentally different about agents having native/local access to structured context?

Claude knows your codebase. But it has no clue “how humans actually interact with your product”.

What if it could… 

I have been thinking about what it unlocks for test automation:

1. Identify untested behaviours and add tests
2. Understand where users are struggling -> check for UX frictions (is there a big JS being loaded slowing down the page? A localization bug?)
3. Capture user segments, write realistic test fixtures that emulate them

All of it - prioritized by usage signals - frequency, time spent etc.

Beyond testing - it could even catch errors and raise PRs to fix, without you needing to monitor and tell “what to fix”, or even come up with product improvement ideas!

This is why instrumentation is one of the best “investments” teams can do today as they move to agentic coding. Historically, this used to be a prohibitively expensive task since you had to:

1. Decide which events to track
2. Decide Metadata and slicing dimensions for each event (user.role, plan, region, etc.)
3. Get everyone aligned on naming and vocabulary
4. Add instrumentation in the right places
5. Repeat after every PR and every feature

Even disciplined teams struggled to maintain this process. And after all that work, someone still had to stare at dashboards and decide what mattered.

But agentic coding changes the economics. All of the above can now be a background task done autonomously - for each PR - by Claude.

What would you do - if Claude could learn user behaviour of your app and use it as a feedback loop?

Product usage driven feedback loop for testing

Claude knows your codebase. But it has no clue “how humans actually use your product”.

What if it could… 

I have been thinking about what it unlocks for test automation:

1. Identify untested behaviours and add tests
2. Understand where users are struggling -> check for UX frictions (is there a big JS being loaded slowing down the page? A localization bug?)
3. Capture user segments, write realistic test fixtures that emulate them

All of it - prioritized by usage signals - frequency, time spent etc.

Beyond testing - it could even catch errors and raise PRs to fix, without you needing to monitor and tell “what to fix”, or even come up with product improvement ideas!

This is why instrumentation is one of the best “investments” teams can do today as they move to agentic coding. Historically, this used to be a prohibitively expensive task since you had to:

1. Decide which events to track
2. Decide Metadata and slicing dimensions for each event (user.role, plan, region, etc.)
3. Get everyone aligned on naming and vocabulary
4. Add instrumentation in the right places
5. Repeat after every PR and every feature

Even disciplined teams struggled to maintain this process. And after all that work, someone still had to stare at dashboards and decide what mattered.

But agentic coding changes the economics. All of the above can now be a background task done autonomously - for each PR - by Claude.

What would you do - if Claude could learn user behaviour of your app and use it as a feedback loop?