u/ElegantConnection737

Earlier I talked about the OQOAI-PBDR project, which allows you to combine idle GPUs on office workstations into a decentralized AI cluster, distributing requests between nodes and choosing the most optimal one.

This week, another update was released in which the OQOAI-PBDR orchestrator received cloud routing functionality.

Under the hood is a lightweight and open-source core based on literally 3 source files of ~1000 lines each, with compiled versions for Windows and Linux supporting Nvidia and AMD GPUs -the entire binary takes about 10 MB. It allows, taking into account dozens of parameters such as:

• GPU and CPU load,
• available VRAM,
• GPU temperature,
• request queue length,
• inference time,
• cold start of the model,
• currently loaded model,
• network latency,
• node idle time,

to make intelligent routing decisions.

Now new criteria have been added to this list:

• economic efficiency — comparing the cost of local generation versus cloud costs,
• model availability in the cloud,
• data privacy - PII and medical data remain local,
• time of day - the cloud may be cheaper at night,
• model size - requests for large models are routed to the cloud.

The system automatically decides: send the request to the local GPU cluster or to one of 40+ cloud providers (OpenAI, Anthropic, Google, Azure, AWS, DeepSeek, Together AI, and others).

Everything works through a unified OpenAI-compatible API - your applications (Open WebUI, N8N, LangChain, LlamaIndex) require no changes.

An interesting detail: to calculate the cost of local generation, the system collects the power_max parameter from the GPU (maximum power consumption) and uses it to compute electricity costs. This allows real-time comparison: what is cheaper - running generation locally or sending the request to the cloud.

Test results:

• Cloud requests are 3.4x faster for small requests (0.85 sec vs 2.9 sec locally)
• Hybrid routing delivers up to 60% cost savings compared to full cloud deployment
• Model cold starts are reduced by 80.7%
• Routing decision takes <5 ms

More details about the research results can be found in the scientific publication on the PBDR v2.0 architecture:

http://doi.org/10.17513/doi.27

The project is completely open-source under the MIT license. Source code and documentation are available in the repository

https://github.com/oqo-ai/OQOAI-PBDR

Leave your questions and use cases in the comments — I'll be happy to discuss them!

We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

• Internal RAG services can prioritize low latency.
• Development workloads can prioritize output quality.
• Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

• OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)
• Admin dashboard with real-time metrics
• Auto-discovery and network scanning
• Remote config management for nodes, groups, or the whole cluster
• Experimental binary builds for Windows and Linux

Roadmap highlights:

• ComfyUI/AUTOMATIC1111 image/video generation
• Grafana/Prometheus integration
• AI-assisted semantic routing (auto-model selection)
• Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?