Deep Dive: How we built a multi-agent orchestration mesh with 4 task-specific SLMs to power our Shopify AI agent
Hey everyone, one of the founding engineers at Yappi here 👍
When we first started building our Shopify AI chatbot, we did what most devs do: we spun up a simple LangChain pipeline, hooked it up to GPT 4.0, threw a basic RAG system on top of our product catalog database, and called it a day.
It worked... okay-ish. But we quickly hit a wall. In e-commerce, generic chatbots don't drive conversions. A standard RAG pipeline will spit out a product description, but it doesn't know WHEN to recommend a product, HOW to handle buying hesitation (objections), or how to capture real-time storefront behavior (e.g., a customer lingering on a checkout policy page).
So, we tore it all down. Over the last few months, we've built a highly distributed, multi-agent orchestration mesh powered by 4 custom-fine-tuned Small Language Models (SLMs) and a real-time behavioral telemetry loop.
Here is a look under the hood at our complete production pipeline:
- Real-Time Behavioral Signal Processing
Unlike standard chat systems where the LLM only sees the TEXT the user types, our orchestrator receives a continuous event stream of the customer’s browser behavior.
We monitor:
- Dwell time on specific product images
- Scroll velocity and depth on sizing charts
- Cart events (additions, removals, quantity changes)
- Click trails showing what category pages they navigated prior to launching the chat
- These behavioral signals are ingested via a low-latency queue, normalized into numerical feature vectors, and injected directly into the prompt context window as specialized temporal state tokens. If a user says "Is this in stock?" while hovering on a red hoodie, the system already knows they mean the Red Hoodie in Size L without needing to ask.
- The Multi-Agent Orchestration Layer
At the heart of the system is the Coordinator Agent. Written as a highly optimized state machine, the Coordinator controls the multi-stage reasoning chains (using a modified ReAct/Chain-of-Thought approach). It doesn't try to answer the query itself. Instead, it decides how to route the user's input and store state across our cluster of specialized, task-oriented Small Language Models (SLMs).
By using smaller, hyper-specialized models instead of one massive monolithic model, we reduce latency down to double-digit milliseconds and keep inference costs manageable.
- Our 4 Proprietary Task-Specific SLMs
We fine-tuned four separate open-weights base models on our proprietary Shopify interaction dataset. Each model does exactly one thing exceptionally well:
- SLM-Intent (3.8B parameters): Determines user intent at a granular level. It distinguishes standard queries ("Where is my order?") from buying signals ("Will this arrive before Friday?") and support inquiries ("How do I return this?"
- SLM-Match (1.5B parameters): Maps vague natural language queries to our shop graph database. If a user asks for "something breezy for a beach wedding," SLM-Match translates this semantic concept into specific product tags, fabrics (linen, cotton), and SKUs
- SLM-Objection (7B Mixture-of-Experts): Fine-tuned specifically for sales objection handling. When a customer expresses hesitation ("this is too expensive" or "I'm worried it won't fit"), this model selects the optimal negotiation strategy (e.g., highlighting warranty, suggesting a discount code, or offering a split-payment option)
- SLM-Upsell (1.5B parameters): Analyzes the current cart value, customer session history, and sentiment data to calculate the exact millisecond to trigger an upsell or cross-sell recommendation. If triggered too early, it annoys the user; if too late, they checkout. This model predicts the highest probability conversion window
- Custom hybrid retrieval pipeline
To make sure our models never hallucinate product details (which is a fast track to getting a merchant sued), we bypassed traditional out-of-the-box vector databases.
We designed a split-path hybrid retrieval flow that enforces strict catalog validation constraints prior to generating a response:
- Dense Retrieval: We embed the merchant's catalog using a custom-fine-tuned embedding model that understands e-commerce terminology
- Sparse Retrieval (BM25): We run lexical searches to capture exact model numbers, colors, and sizes
- Graph Constraint Solver: The results of the dense and sparse searches are merged, then fed into a constraint solver which queries the Shopify Admin API in real-time to check stock levels, active price adjustments, and checkout rules
- Cross-Encoder Reranker: A final cross-attention step reranks the validated products using the telemetry feature vectors from the user's active session
- Multi-Stage Reasoning & Synthesis
Once the retrieval pipeline returns the valid catalog data, the Coordinator routes all inputs (Behavior features, Intent class, Objection strategy, Product matches, and Context) into a final synthesis agent. This agent generates the response stream using a dynamic, self-correcting logic chain. Before the response is sent back to the customer over a persistent WebSocket connection, a safety filter (built on a custom LLaMA Guard wrapper) checks the output for price mismatches, policy compliance, and toxicity.
Why we built it this way (and why it matters):
It would have been 100x easier to just build a standard wrapper around GPT 4.0. But e-commerce conversion rates live and die by latency and relevance. Monolithic models are too slow, too generic, and lack the real-time context of e-commerce user behavior. By distributing the workload across tiny, specialized SLMs and coordinating them through a lightning-fast event bus, we've managed to achieve sub-second response times while driving actual double-digit conversion rate increases for our merchants.
Would love to hear from other folks building in the e-commerce AI space!
* How are you all handling zero-shot product matching when catalogs update in real-time?
* Are you seeing better results with fine-tuned SLMs vs heavily prompted frontier LLMs?