u/ItemBusiness4500

I've been looking at a lot of support dashboards I’m starting to think deflection is the most dishonest metric in customer support.

We celebrate Bot Deflections and AI Resolutions as wins. We show the VP of Support a chart showing 40% of tickets were handled by a bot and they celebrate the cost savings.

How many of those deflected tickets are going to be opened again and again? How many customers finished a chat and come back 20 minutes later because the bot gave them a generic answer that didn't solve their issue?

We’re optimizing to close the ticket instead of solving the damn problem.

If a customer has to hit the help button three times for the same problem then it isn't a deflected ticket.

How do you track if a bot actually solved something, or if it just successfully shut the customer up for a few hours? Do you care about re-opens, or is deflection rate the only one that counts?

And at what point does a resolved ticket become a repeat problem that should have a permanent solution?

Here's what changed. It's same data, entirely different shape:

Before:

"Here are 15 FAQs to write, sorted by frequency."

After:

"This billing question repeats 420x/month, costs ~$3,200/month in agent time, and Billing is the probable owner."

We shifted from ranking questions plus cost and started attaching recurring cost and a probable owner to each one. The data didn't changed, but because "write this FAQ" lands in a content backlog, And "this question is costing you $3k/month and Product owns it" gets forwarded to a VP.

Both versions are opinionated. But the new one makes the invisible repeats visible and routeable, which is what support leaders keep telling us they want. They need a way to hand issues back to the team that can end it.

Honest question for anyone who's been on either side of this. Is adding cost and owner more useful, or does it start to feel like the tool is overstepping? We're still calibrating that line.

Sharing what works for me, not claiming it’s the only way.

1. Don’t verify the transcript. The transcript is where “claimed done” lives. It’s self-reported, and if the run was long enough to compact, the summary can quietly drop the one constraint that mattered — so the agent finishes confidently against a goal that’s subtly not yours anymore. Reading it tells you what the agent thinks it did, which is the least reliable thing in the system.

2. A run is only “done” when a check you didn’t write passes. Make every run end in a checkable artifact — a PR, a diff against a spec, a green pipeline — and gate on the artifact, never the narration. “Done” means a check passed. It never means the agent typed done.

3. Run the whole test suite, not the test the agent picked. The classic silent failure: new feature works, old thing broke, agent only ran the new test. If the agent gets to choose which tests prove its own work, that’s your hole. Close it.

4. Writer ≠ checker. The session that wrote the code is the worst reviewer — it shares its own blind spot. A second agent with a clean context, whose only job is to reproduce the verification steps and hunt for regressions, catches things the author can’t. Different context, no stake in the claim.

5. Content gates, not just pass/fail tests. CI can fail a PR for more than a broken build. Mine fails if the PR body is missing required sections, or if the copy makes a claim that isn’t in an approved registry. Anything you can write down as a rule, you can fail the run on.

On continuous loops specifically: the part that makes me nervous is compaction. A continuous loop forces you to bet on the agent resuming exactly where it left off after the context gets summarized, and that’s a gamble — there’s no guarantee it picks up the same intent. Discrete jobs sidestep it: each one small enough to finish inside a single context window, each ending in a pass/fail artifact, and the next job rebuilds its state from that artifact (a PR, a checklist file) instead of from the conversation. If resume needs the transcript, you’ve already lost.

What this doesn’t fix: it won’t catch a bug that’s wrong-but-passes-tests — your gate is only as good as your suite. It costs CI time (full suite + a second reviewer pass). And it’s more upfront wiring than just letting a loop run. For me the trade is worth it: a failure shows up as one red pipeline instead of an afternoon of re-verifying after the fact.

Curious what others gate on besides tests — anyone failing runs on diff size, perf budgets, or schema checks?

You identify a high-volume, high-cost repeat question and you prove it’s a product gap or a policy issue. You present the data. And then... nothing happens.

I've been working on a way to quantify this and not just by volume, but by real agent hours spent on unresolved questions. The idea is to separate the "documentation fixes" from the "gaps" (the things no help article can fix because the process itself is broken).

But I’m hitting a wall on the implementation side.

Even with a perfect audit that shows exactly where the money is leaking, the "insight-to-action" gap remains. A list of problems doesn't equal a change in priority.

I'm curious to hear from anyone running a support org: How do you actually turn a support audit into a product or policy change? Does the data actually move the needle, or does it just become another tab that everyone looks at but nobody acts on?

I've been playing around with some analysis on Zendesk exports.

Most of us know which topics have high volume, but it is much harder to prove the actual cost of the questions that don'thave a good Help Center article. I've been working on a way to automate the identification of these "blind spots."

The logic is simple:

Identify questions with high volume and high agent touch-time.

Check if a corresponding article exists in the Knowledge Base.

If no article exists, look at how agents are currently solving it in the tickets to draft a "benchmark" answer.

I wanted to avoid the usual "AI-powered" hype. No LLMs are used for the analysis, so there's no risk of the model making up answers. All answers are pulled from actual agent resolutions found in the CSV.

For those of you managing large teams: If you could get a report that ranked your top 10 most expensive "missing" articles, would that actually change your roadmap, or is it just another thing to add to a meeting agenda

Hey everyone,

I’ve spent way too much time in the past exporting Zendesk ticket data to CSVs, throwing it into Excel, and trying to manually group support tickets just to figure out what new FAQ articles we needed to write. It was mind-numbing and usually took days of reading through individual tickets.

Most "AI deflection" tools I looked at wanted me to install a widget or buy an expensive subscription before they'd even tell me if we could deflect anything.

So I spent the last few weeks building a standalone pipeline to just do the analysis part automatically. I’m trying to validate if this is a problem other support leaders actually care about, or if I over-engineered a solution to my own annoyance.

How it works right now: You upload a CSV export of your Zendesk/Freshdesk tickets. The script chews through it and spits out a one-time report that shows:

1. Ranked Cost of Unresolved Repeat Questions: This is the core output. It finds your top repeat questions that currently have no consistent resolution in your ticket history, and surfaces the cost of each one — how many tickets, how many agent-hours, and how much that's burning each month. You're not looking at a ranked list of topics; you're looking at a ranked list of open bleeding wounds with a dollar figure attached.
2. Draft Answers (where they exist): For questions where your agents have been solving the issue consistently, it drafts a publish-ready answer based on how they're already doing it (including source ticket quotes so you can verify it's not hallucinating). These are ready to copy into your Help Center.
3. "Blind Spots" — Unresolved at Scale: For the high-cost questions where no consistent resolution exists, instead of trying to invent an answer, it flags these explicitly: broken process, missing policy, product gap. This is what separates "fix your docs" from "fix your product."

It exports all of this into a PDF you can hand to leadership, and a raw JSON/CSV file you can give to engineers if they want to train a bot.

My questions for this community:

1. Is this actually a major pain point for you? How are you currently figuring out which unresolved repeat tickets are costing you the most?
2. Does ranking by cost of the unresolved question (rather than just ticket volume) change how you'd prioritize what to fix?
3. Does the "Blind Spot" framing (separating missing documentation from broken processes/product gaps) make sense, or is that distinction too subtle to be useful?
4. Would you pay a one-time fee for a deep-dive report like this (no subscription), or do you prefer tools that just live inside Zendesk permanently?

I have a working prototype if anyone is crazy enough to want to test it on their own ticket export (I can share the link in the comments or DM, not trying to spam the main post).

Really appreciate any brutal honesty you can throw my way!

I built a tool that pulls apart messy support queues to fix this. It isolates the actual questions that have not been resolved, identifies what can genuinely be self-served, flags tickets where customers are just coming back because the root issue was ignored, and extracts the exact language your customers use (which is a goldmine for SEO and product marketing).

Right now, I have only run this on synthetic data. To make the system truly bulletproof, I want to partner with a team that has real, messy, high volume customer tickets.

If you run your support through Zendesk, Intercom, Freshdesk, or a similar platform, and you have a decent volume of tickets, I would love to collaborate. I am looking for an anonymized CSV export from the last 30 to 60 days. All PII stripped out.

In exchange, I will run your data through the system and hand over every single finding for free. You get the full breakdown of your repeat clusters, the true self-serve vs human split, and all the raw customer language insights to hand straight to your marketing team.

No pitch at the end. I just need real-world volume to refine my model, and you get a completely free deep dive into your support queue.

If you have the volume and are interested in a data-for-insights partnership let me know.

Take a look at the landing page.

juancanfield. com/systems/support-ticket-deflection/snapshot

I built a tool that pulls apart messy support queues to fix this. It isolates the actual questions that have not been resolved, identifies what can genuinely be self-served, flags tickets where customers are just coming back because the root issue was ignored, and extracts the exact language your customers use (which is a goldmine for SEO and product marketing).

Right now, I have only run this on synthetic data. To make the system truly bulletproof, I want to partner with a team that has real, messy, high volume customer tickets.

If you run your support through Zendesk, Intercom, Freshdesk, or a similar platform, and you have a decent volume of tickets, I would love to collaborate. I am looking for an anonymized CSV export from the last 30 to 60 days. All PII stripped out.

In exchange, I will run your data through the system and hand over every single finding for free. You get the full breakdown of your repeat clusters, the true self-serve vs human split, and all the raw customer language insights to hand straight to your marketing team.

No pitch at the end. I just need real-world volume to refine my model, and you get a completely free deep dive into your support queue.

If you have the volume and are interested in a data-for-insights partnership let me know.