r/EnterpriseArchitect

We're in the middle of building a business case for moving off a legacy monolith and every conversation we've had runs straight to licensing and implementation without touching the stuff that blows the budget once you're in the weeds.

ERP integration is the main one, particularly the bidirectional sync for order management and inventory where estimates have ranged from 6 weeks to 6 months depending on who you talk to, and every platform we're looking at (commercetools, SAP Commerce Cloud, SCAYLE, Spryker) has given us figures in wildly different ranges where the delta seems to come from how much of that integration layer is already built and how much is custom scope.

Data migration is a separate headache because our catalog is 4 years of accumulated schema decisions that don't map cleanly to anything off the shelf, and the QA coverage budget for a parallel-run period is listed as "TBD" in every proposal we get back.

That's about where our visibility runs out, so what hit your budget that wasn't in the original scope?

Disclosure: I’m the CPO at SearchBlox and this is our product. It’s free to download and run.

This came out of enterprise deployments where a GPU simply wasn’t on the table — air-gapped networks, regulated data, and procurement cycles measured in quarters. We got tired of telling those customers “wait for hardware,” so we built for the CPUs they already had racked. Posting here because I think CPU-only is more viable in 2026 than most people assume, and this is the right crowd to poke holes in it.

What it is
• One self-contained binary. curl -fsSL https://inference-server.searchblox.com/install | sudo bash → 2–10 min, service starts, API key generated at /etc/searchai/server.properties. (If piping to sudo bash makes you twitch — fair — the binaries and installer script are downloadable directly with published sha256.)
• One OpenAI-compatible endpoint at :8081/v1 for everything: chat, grounded RAG, function calling / MCP, guaranteed JSON via response_format, vision + video, speech-to-text, text-to-speech including file-based voice cloning. Change base_url, keep the rest of your OpenAI SDK code.
• Ships a 4B text+vision model (~3.4 GB) plus ASR/TTS (~5 GB extra). A 2B option for lighter hosts.
• 380 tested prompts across 13 industries in the built-in console — each one copyable as Python, cURL, or JSON. This is the part I’d actually use first; it’s a much faster way to find out whether a 4B is good enough for your task than writing your own harness.
• Cluster mode built in: model-aware, load- and cache-aware routing. Any node accepts any request, different nodes can serve different models. Scale by adding hosts.
• HTTPS and Bearer auth on by default, usage.total_time_ms on every response.

Where CPU loses, and I’d rather say it than have you find out

Long open-ended generation, large models, and many simultaneous token streams. If that’s your workload, buy the GPU — there’s a one-flag BACKEND=cuda add-on for exactly that. The argument for CPU is that grounded document Q&A, summarization, extraction to JSON, classification, function calling, and transcription — which is most of what enterprises actually deploy — fit inside CPU latency budgets with no accelerator to reserve, no cold starts, and no second system to secure.

Linux x86_64/arm64, Apple Silicon, or Docker on Mac/Windows. Free, no license key, support plans optional.

https://inference-server.searchblox.com

Happy to answer anything in the comments — including “why not just llama.cpp + whisper.cpp + a reverse proxy,” which is a completely reasonable question.