Disclosure: I’m the CPO at SearchBlox and this is our product. It’s free to download and run.

This came out of enterprise deployments where a GPU simply wasn’t on the table — air-gapped networks, regulated data, and procurement cycles measured in quarters. We got tired of telling those customers “wait for hardware,” so we built for the CPUs they already had racked. Posting here because I think CPU-only is more viable in 2026 than most people assume, and this is the right crowd to poke holes in it.

What it is
• One self-contained binary. curl -fsSL https://inference-server.searchblox.com/install | sudo bash → 2–10 min, service starts, API key generated at /etc/searchai/server.properties. (If piping to sudo bash makes you twitch — fair — the binaries and installer script are downloadable directly with published sha256.)
• One OpenAI-compatible endpoint at :8081/v1 for everything: chat, grounded RAG, function calling / MCP, guaranteed JSON via response_format, vision + video, speech-to-text, text-to-speech including file-based voice cloning. Change base_url, keep the rest of your OpenAI SDK code.
• Ships a 4B text+vision model (~3.4 GB) plus ASR/TTS (~5 GB extra). A 2B option for lighter hosts.
• 380 tested prompts across 13 industries in the built-in console — each one copyable as Python, cURL, or JSON. This is the part I’d actually use first; it’s a much faster way to find out whether a 4B is good enough for your task than writing your own harness.
• Cluster mode built in: model-aware, load- and cache-aware routing. Any node accepts any request, different nodes can serve different models. Scale by adding hosts.
• HTTPS and Bearer auth on by default, usage.total_time_ms on every response.

Where CPU loses, and I’d rather say it than have you find out

Long open-ended generation, large models, and many simultaneous token streams. If that’s your workload, buy the GPU — there’s a one-flag BACKEND=cuda add-on for exactly that. The argument for CPU is that grounded document Q&A, summarization, extraction to JSON, classification, function calling, and transcription — which is most of what enterprises actually deploy — fit inside CPU latency budgets with no accelerator to reserve, no cold starts, and no second system to secure.

Linux x86_64/arm64, Apple Silicon, or Docker on Mac/Windows. Free, no license key, support plans optional.

https://inference-server.searchblox.com

Happy to answer anything in the comments — including “why not just llama.cpp + whisper.cpp + a reverse proxy,” which is a completely reasonable question.

Been wrestling with this. Most RAG setups I see treat the agent as the center and search as something you wire up underneath — custom retrieval glue, re-ranking you maintain by hand, brittle handoffs.

The MCP approach inverts it: expose search as a tool (hybrid BM25 + vector, citation grounding, KG context all behind one interface) and let any agent just call it. The agent stops owning retrieval logic and starts treating search like any other capability.

What I like: governance and access control stay in the search layer, so an agent can’t accidentally leak across collections — matters a lot for regulated/air-gapped setups.

What I’m unsure about: are we just moving the complexity, not removing it? And does tool-calling latency kill it for multi-hop reasoning?

For those running agentic retrieval in prod — are you exposing search via MCP, or still building bespoke pipelines? What broke?

(Disclosure: I work on an enterprise search platform, so I’m biased toward the tool-first view — genuinely want to hear the counterargument.)

The Problem: Enterprise AI Is Still Too Complex

If you’re already using Google Cloud, you have access to some of the most powerful AI services in the world — Vertex AI, Gemini, and Google Workspace.

But building a real enterprise AI solution still looks like this:

• Stitch together pipelines (ETL → embeddings → vector DB → LLM)
• Manage multiple tools and APIs
• Handle permissions separately
• Build orchestration logic manually
• Maintain and scale everything

Turning Intent Into Action

From matching keywords to results, to enabling conversational search to fetch summarized answers. Human expectations of search keep evolving.

And now, the needle has moved again.

Users now expect search to understand what they mean, find the right information across systems, and help them take the next step.

Whether someone is looking for an HR policy, troubleshooting a customer issue, choosing the right product, or completing an internal workflow, the expectation is the same: don’t just show information - make it useful.

This gap — between retrieval and understanding — is precisely what agentic search is designed to close.