Integrating standard operation procedures with agentic AI workflow
Hello guys, me and my team have been building an agentic workflow to answer customer questions (rn in langgraph).
The use case goal is to answer ALL customer support questions. We use a RAG (~1500 different documents) and tool calling to access databases with the client information.
The workflow is basically this:
- Use the question and the tools description as context to generate a plan (The plan will be something like call tool A with argument X)
- Execute the plan
- Use the question and tools output to generate an answer
This is already in production but with a HIL that validates the AI generated answer. Right now, in 50% of the tickets the AI answer is correct (no changes are made to it by the HIL). But the team wants to increase this percentage, so we started to look for differences in the final answer generated by the HIL and the AI answer.
When analyzing the differences, we understood that there is a sort of shadow knowledge base based on the customer support team tacit knowledge. E.g, when the question is about a specific topic, you must ask for a specific document (there is no doc in the knowledge base with this information the customer support simply knows it).
Therefore, we mapped this tacit knowledge into standard operation procedures (the detailed SOPs cover ~10% of tickets) and now we are wondering how to add it to our current workflow. Does it make sense to create an agent that decides whether to follow a detailed SOP or the "normal" workflow?
I was wondering if anyone has experienced this type of problem and how did they overcome it?
Thanks!!