We have built a ticketing system where the AI runs on every single message. The AI does intent classification, sentiment extraction, ticket title generation, commitment detection and RAG-based answers. When you process that kind of volume the model selection matters.
In theory picking a model should be easy. There are sites like Artificial Analysis that give you model cards with quality, speed and cost.. In practice generic benchmarks are almost useless for our proprietary workloads. Knowing how a model scores on GPQA does not tell me much about how it handles our RAG setup or how it interprets our highly tuned intent-classification prompts.
So we built an evaluation framework called Project Statistix. It lets us define our workloads, such as intent classification, categorization, summarization and RAG Q&A and run them across any model and any inference provider.
Here is what surprised us:
The former kings, such as the GPT-4.1, the GPT-4.1-mini and the GPT-4o-mini are still fast. Looking at the cost-to-speed spectrum they are often no longer the optimal choice for our conversational ticketing system workloads.
Another surprising thing: when we reviewed the test outputs we found edge cases in which all the top models disagreed with what we had defined as the ground truth for our ticketing system. When we dug in the models were not wrong our prompts were ambiguous.
The benchmarking did not just help us models, for our conversational ticketing system. It stress-tested our prompts. Improved them.
We are hoping to open-source the Project Statistix soon. We are curious if anyone else has built internal evaluation frameworks and what abstractions worked for you for your conversational ticketing system.
Salesforce buying Fin for about $3.6B feels like a very clear bet that customer support will be the first big, measurable enterprise AI-agent market: https://www.reuters.com/business/salesforce-buy-fin-about-36-billion-2026-06-15/
What’s interesting to me is the timing. AI agents are still fuzzy in many enterprise use cases, but support is one of the few places where the ROI is brutally visible. Say tickets resolved, handle time reduced, cost to serve lowered. Fin claims an average support-volume resolution of around 76%, which is exactly the kind of metric Salesforce can sell into Service Cloud accounts.
The risk is that Fin’s value partly came from being a focused, independent product. Inside Salesforce, it could either become the agent layer Service Cloud badly needs, or get absorbed into the usual big-platform maze. My gut reaction is smart acquisition, expensive but strategically coherent.. 😃
Okay so I need to talk about this a bit. I am also really looking for how others are solving this problem.
We support our customers over Slack. At first it was great. Customers loved it it felt personal. We were able to respond quickly.. As we have grown it has become a bit of a mess. I am not sure how to fix it.
Here is what is actually happening day to day:
>Nobody knows what is open and what is done
A customer asks something in a channel. Someone from our team replies. Then the thread just stops. Was the issue resolved? Did the customer get what they needed? I have no idea. I am constantly going back through threads wondering. Did we actually close this out?
>Things fall through the cracks all the time
We have than 30 customer Slack channels now. A message comes in. Everyone assumes someone else will respond.. Nobody does. The customer follows up two days later. It is embarrassing. I hate it.
>No ownership of anything
There is no way to say "hey this one is yours" inside Slack. It is a thread. So either one person ends up handling everything or nothing gets handled. Neither option is good.
>No metrics all
I have no idea how fast we are responding to customers. I have no idea which customers are waiting the longest. I cannot tell if we are getting better or worse. My head of support is asking me for numbers. I am just making things up at this point.
>The Zendesk/Jira thing is killing
When something needs to be escalated. Like a bug or a feature request. The customer has talked to us for 20 messages on Slack and now we have to go recreate the whole thing in Zendesk or Jira from scratch. That is a waste of time.. Half the context gets lost in translation.
>We are basically doing triage in our heads
Every morning someone on the team manually scans through every channel to figure out what needs attention. That is a person spending real time just reading Slack to figure out what we missed. It is not scalable. It is burning people out.
I know Slack was not built for customer support. But we are not ready to move customers off it. They like it. So do we honestly. We just need some structure around customer support on Slack.
How are you all dealing with customer support, on Slack? Is there a workflow, a tool something I am missing? I would love to hear what is actually working for teams our size.
We work with CS teams managing anywhere from 30 to 300+ customer Slack channels. I want to share the exact sequence of what breaks as you scale, because it's remarkably consistent across companies.
10-20 channels: Everything's fine. Your team can scroll through them. You roughly know what's open. No tools needed.
20-50 channels: First cracks. You start missing threads. Someone asks a question at 2pm, nobody sees it until the next morning. You try emoji reactions to mark things as "needs attention" vs "resolved." It kind of works.
50-100 channels: Full breakdown. Your team spends the first 30 minutes of every day scrolling through channels to figure out what needs attention. Nobody has a clear picture. Things fall through the cracks weekly. Customer complaints about slow responses start.
100+ channels: Chaos. You have no idea what's open, who's handling what, or what your actual response times are. You're reactive to whoever is loudest. Quiet customers with real problems get ignored. You start losing deals over support quality.
The fix at each stage is different:
At 20-50, you need a triage channel, one place where all conversations across all customer channels show up. That's the minimum.
At 50-100, you need SLA tracking on top of that triage view. Otherwise you're still guessing about what's urgent.
At 100+, you need automation, AI to handle repetitive questions, auto-categorization, and routing rules so the right person sees the right request.
Where are you in this sequence? And what have you tried so far?