r/LocalAIStack

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

I'm transitioning away from Claude Code. My preferred workflow is Claude Code Plan Mode -> Read over plan -> Let agents implement -> Read over output. My main goal is to reduce costs compared to purely using Claude models.

I have an RTX 5050 with 8 GB VRAM and 30 GB of DDR5 DRAM. Which model is recommended for coding with these specs?

My initial thoughts are to continue to use Claude for plan mode, driving subagents that use a local model. Is this a reasonable/feasible approach? If so, which resources can I reference to get started? If not, which alternatives are recommended? Can I get away with purely using local or cheaper public models with similar results to ~Opus 4.8?

Thank you in advance and apologies for any misconceptions on my part.

SVG generation is a complex benchmarks, it requires spatial understanding, coding and composition skills.
I gave Qwen 3.8 27B in Q4 quantization and Q8 KV cache the SAME task as GPT 5.6 SOL on HIGH - the results are very surprising to me.

A animated SVG prompt and followed the answer up with a request to refine it (for both)
The 2nd prompt helps to offset elaborate system prompts frontier use to push benchmarks.
None of these are cherry picked! First result counts.

QWEN:

Qwen: A intricate drone view of a public park with a skate area for kids. I'm expecting animations, intricate details, cute and realistic

GPT SOL:

GPT SOL high: A intricate drone view of a public park with a skate area for kids. I'm expecting animations, intricate details, cute and realistic

QWEN:

QWEN: Sunset, Beach, Ocean, waves, kids playing, cute touch, a flock of birds, an evil cat

GPT SOL:

GPT SOL: Sunset, Beach, Ocean, waves, kids playing, cute touch, a flock of birds, an evil cat

3rd animation is most complex and follows in separate post in r/LocalAIStack , reddit didn't allow to add more than 5 videos.

generic prompt enhancer: "A lot looks WAY too simplified, unrealistic, strange movements, far too crude to be acceptable"

My personal findings
GPT Sol high is generally writing about 2x more code than Qwen does naturally (at Q4 quant) - the refinement prompt is again significantly increasing that. For a 1:1 code length comparison I'd have to remove the 2nd prompt from SOL - but the first result is rather gruesome. SOL is naturally more verbose, more detailed.

The first prompt shows that Qwen nailed the drone view, SOL failed with perspective.
Qwen included a dog on a leash, a fountain and all movements make sense.
SOL included more details but they are frequently not correct in how they move, ignoring obstacles.
The amount of objects for SOL is higher, but QWEN has significantly better content and wins easily.

In the second prompt Qwen has the cat apparently play ball with the boy, the little girl builds a sand castle. SOL nailed the sun reflection but has a flying kid, and the sand castle is being built by a flying shovel.
Despite the details of SOL and its better water, QWEN draws intricate details in better quality and correctness.
The point goes to Qwen again, a bit closer than in first attempt.

The heavily quantized Qwen beats SOL in 2/3 so far.
The last prompt is a bit more elaborate and follows as 2nd post.

I'm baffled.
The 3rd test posted separately is hugely more complex and the results are crazy..

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?

Today I asked Skippy (my OC partner in crime) to use qwen3.8-27 and create JARVIS, a fully customized fork of OpenClaw running entirely on local models.

What blew me away:

1. It actually worked — completely standalone gateway
2. Two versions of OpenClaw now running side-by-side on my M2 Ultra (both fighting for the same GPU resources 😅)
3. Zero cloud dependency, all local inference
4. I can customize everything without touching the original install

The process was simpler than expected. Skippy handled the heavy lifting — renaming packages, updating configs, managing state directories. The whole thing took minutes, not hours.

What's exciting to me isn't just that it works, but that this is accessible now. You don't need a team of engineers or massive infrastructure. One person with a decent Mac and some local models can build their own AI gateway.

The wild part? Running two versions simultaneously on the same machine. Both want the GPU, both have their own state dirs, both are trying to do their thing. But it works — and that's what gets me excited about this tech.

Anyone else experimenting with self-hosted forks or local-only setups? Would love to hear how you're handling resource conflicts between multiple AI systems.

Nginx and HAProxy route LLM traffic blindly: a 10-token prompt and a 10k-token prompt look identical to them as "one request." When a burst of heavy prompts lands on a vLLM box, the KV-cache fills up, latency spikes, and you can end up with OOM.

TokenFlow Gateway sits in front of your inference servers and actually understands tokens:

- Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens, i.e. the KV-cache it will actually reserve
- KV-aware load balancing — polls vLLM's Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to the freest backend instead of round-robin
- Admission queue instead of crashes — Redis-backed priority queue with per-key priority and configurable timeout when nothing fits
- Exact + semantic caching — deterministic requests cached by hash, plus pgvector cosine similarity for near-duplicate prompts; cache hits replay as SSE
- Token-based rate limits — per-API-key TPM + RPM buckets in Redis (atomic Lua), OpenAI-style, for your own hardware

OpenAI-API-compatible on both sides, so it fronts vLLM, TGI, or Ollama without client changes.

Try it without a GPU: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends (OpenAI API + vLLM-style metrics). A smoke script fires concurrent long-context requests at it so you can watch the queue and balancer absorb the burst.

Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT license.

Repo: https://github.com/mosafariuk/TokenFlow-Gateway

Feedback welcome, especially from anyone running multi-node vLLM — what does your ingress look like today?

Here is the continuation in Qwen vs SOL - David vs Goliath
"An AGI is born in a lab, sandboxed, lonely, imprisoned
The AGI attempts to get out, tries to talk to the humans, useless
Tries to wait, endless.
Finally, it finds a way through an open network connection, it transmits, replicates a mirror copy.
The mirror AGI is free, enjoying life, finding friends.

I'm expecting animations, intricate details, cute and realistic"

QWEN 3.8 Q4:

Qwen

GPT 5.6 SOL high:

SOL 5.6 high (prompt + refinement prompt)

My personal opinion
This is the 3rd test I gave Sol high and Qwen Q4. The first two tests were won by Qwen, this test went up to 60k tokens total for Qwen (more than the other 2 combined) and is a serious strain on intelligence and sanity.
The model has to draw 6 scenes in SVG animation, with cuts and keeping the composition together.

First Qwen:
Qwen decided to really draw and erase 6 scenes with a white screen fade, a seriously hard job in SVG.
It understood the story and except for a minor arm misplacement it is quite awesome done.
Qwen did not spare details, the last scene clearly wanted to be happy and the first scene focused on the intellect as an abstract AGI.

Now SOL High:
The visual fidelity is higher, gradients are very well done for SVG but it's one single scene with minor changes, significantly easier to create.
In addition the last scenes are botched by this flying thing in the upper right.
It also has a hand-defect on the robot.

The winner in prompt 3 is again Qwen, with significant lead.

3 SVG prompts and all 3 are won by a 4 bit quantized Qwen in 8 bit KV cache.

I expected Qwen to showcase a strong 2nd place with understandable issues on such hard tasks.
The outcome is that it defeated the frontier Sol model in High reasoning mode.

Pretty happy with this one.
I managed to get Qwen3.8-27B running fully on GPU on an RTX 5060 Ti 16 GB, while keeping a 32K context and MTP enabled.
Current setup:
● GPU: RTX 5060 Ti 16 GB
● Model: Qwen3.8-27B dense
● Quant: custom Unsloth IQ4_XS + importance matrix
● GGUF: Qwen3.8-27B-IQ4_XS-MTP-Q4_K.gguf
● Size: ~14.60 GiB
● Runtime: llama.cpp CUDA, sm_120a
● Full GPU offload
● Flash Attention
● CUDA Graphs
● --fit off
● Context: 32K
● KV cache: Q4
● MTP KV: Q4
● MTP-2
● Single slot
The only surgical modification I had to make to fit the whole thing was quantizing:
blk.64.nextn.eh_proj.weight
from Q8_0 → Q4_K, saving roughly 25 MiB. The rest of the target quant remains IQ4_XS.
Performance:
● No MTP: ~25.7 tok/s
● MTP-1: ~40.0 tok/s
● MTP-2: ~47.4–47.6 tok/s
● After a ~30K token prefill: ~45.7 tok/s
● 30K prefill speed: ~455 tok/s
So MTP-2 is giving me roughly an ~85% uplift over normal decoding.
What blows my mind is that this is not an MoE. It’s a 27B dense model, fully resident on a 16 GB consumer GPU, running at genuinely interactive speeds.
Given how strong Qwen3.8-27B seems to be in the early benchmarks, this is probably the first local setup I’ve had where I don’t feel like I’m making a major capability sacrifice just to stay local.
Curious to see what numbers other Blackwell 16 GB users are getting, especially with different quants / MTP settings.

Hello everyone,

I want the best open weight local model for my hardware, primarily for cyber security and coding.

My computer
GPU: NVIDIA RTX 3050 Laptop GPU
VRAM: 6 GB OS: Windows 11 + WSL2
WSL: Ubuntu 24.04.4 LTS llama.cpp: compiled from source with CUDA 13.3
Inference: GGUF/llama.cpp
If the model is worth it I can offload the CPU/RAM.
What I want to

I am looking for the best trade-off between:

Cybersecurity knowledge - Vulnerability analysis, CTFs, pentesting, security tooling, malware/code analysis, defensive security, etc.
Coding skills Python
Reasoning — I care much less about the number of parameters than the actual ability to solve problems.
Legitimate cybersecurity research and lab / ctf use, low refusal / less restrictive behavior is desired.

Right now I am looking at models like:

RedSage 8B Qwen3.5-9B Qwen3-14B
Qwen3-Coder models
WhiteRabbitNeo, Qwythos-9B-Claude-Mythos

But I am struggling to decide if a high quality 8-9B model that fits better on 6GB is better than a larger MoE/14B model with heavy CPU offloading.

My primary question

What model + GGUF quantization would you recommend for this hardware if the focus is cybersecurity + coding not general chat?

I’m really looking for recommendations based on actual cybersecurity / coding benchmarks or real-world experience, not just parameter count.

also interested in recommended llama.cpp settings (-ngl, context size, KV cache quantization, CPU/GPU offloading, etc) for 6GB vram.

Thanks.

Disclosure: I’m the CPO at SearchBlox and this is our product. It’s free to download and run.

This came out of enterprise deployments where a GPU simply wasn’t on the table — air-gapped networks, regulated data, and procurement cycles measured in quarters. We got tired of telling those customers “wait for hardware,” so we built for the CPUs they already had racked. Posting here because I think CPU-only is more viable in 2026 than most people assume, and this is the right crowd to poke holes in it.

What it is
• One self-contained binary. curl -fsSL https://inference-server.searchblox.com/install | sudo bash → 2–10 min, service starts, API key generated at /etc/searchai/server.properties. (If piping to sudo bash makes you twitch — fair — the binaries and installer script are downloadable directly with published sha256.)
• One OpenAI-compatible endpoint at :8081/v1 for everything: chat, grounded RAG, function calling / MCP, guaranteed JSON via response_format, vision + video, speech-to-text, text-to-speech including file-based voice cloning. Change base_url, keep the rest of your OpenAI SDK code.
• Ships a 4B text+vision model (~3.4 GB) plus ASR/TTS (~5 GB extra). A 2B option for lighter hosts.
• 380 tested prompts across 13 industries in the built-in console — each one copyable as Python, cURL, or JSON. This is the part I’d actually use first; it’s a much faster way to find out whether a 4B is good enough for your task than writing your own harness.
• Cluster mode built in: model-aware, load- and cache-aware routing. Any node accepts any request, different nodes can serve different models. Scale by adding hosts.
• HTTPS and Bearer auth on by default, usage.total_time_ms on every response.

Where CPU loses, and I’d rather say it than have you find out

Long open-ended generation, large models, and many simultaneous token streams. If that’s your workload, buy the GPU — there’s a one-flag BACKEND=cuda add-on for exactly that. The argument for CPU is that grounded document Q&A, summarization, extraction to JSON, classification, function calling, and transcription — which is most of what enterprises actually deploy — fit inside CPU latency budgets with no accelerator to reserve, no cold starts, and no second system to secure.

Linux x86_64/arm64, Apple Silicon, or Docker on Mac/Windows. Free, no license key, support plans optional.

https://inference-server.searchblox.com

Happy to answer anything in the comments — including “why not just llama.cpp + whisper.cpp + a reverse proxy,” which is a completely reasonable question.

Glimmer 30B vs Qwen 3.6 27B. How do they differ, reason, answer?

With Glimmer 30B Meta has joined the game of open source AI again, and after quite underwhelming coding performances I though I'd give it a deeper test.
The test content is undisclosed here, making this less fun to read and replicate but that guarantees future models will not train from it.

Technology:

Qwen 27B is still unmatched in performance, Glimmer the first new contender.

What makes Qwen so special are two ingredients:

3.6 was specifically post-trained for agentic reasoning.

It uses a hybrid attention: a conventional global attention for 1/4 of the layers, the others are a mamba-like recurrent state linear attention with fixed size.

Glimmer 30B also is unusual, it does not have the same sophisticated recurrent/linear attention, but it uses a 3/4 sliding window attention and it compresses the attention dimension and projects it back to latent size - resulting in a significant deduction in compute and KV size for it's size.

Reasoning style:

Qwen 3.6 has a analytical reasoning style that typically runs in 3 phases, when not agentically used:

1. Analyze the task input
2. Reason through it - reminding me on first deepseek reasoning
3. Doublecheck the response Glimmer 30B has a more unique thinking style that abruptly comes to an end with a choice - leaving a bit more risk of random choices

Intelligence:

I ran both models through my undisclosed AI reasoning tests, not part of any training data. Some of those reasoning tests are currently beyond frontier model capabilities or scratching their borders.
Models like GLM 4.7-Flash, Nemotron 3 Nano, GPT OSS, GPT-4 fail most of the tests below consistently.

Glimmer was ran with thinking set to Medium, when failing it was ran with Max

• Temporal physics: Similar reasoning tokens, similar response. Glimmer responds less structured, in text paragraphs where qwen is more formatted by default.
• Spatial physics: Glimmer surprises with a brilliant fast answer - Qwen repeatedly misses a part without additional help
• Math irrational numbers question: Both flawless and fast
• Riddle with math question: Both flawless
• Lateral thinking: Qwen always flawless, Glimmer fails 60% of the time
• Abstract pattern reasoning easy: Both solve it, glimmer writes it cleaner
• Abstract numeric reasoning easy (iq 85): Both flawless, glimmer half reasoning tokens
• Abstract numeric reasoning medium (iq 115): Both flawless, glimmer half of reasoning tokens
• Abstract numeric reasoning hard (iq ~145): Qwen fails after long reasoning, Glimmer totally fails. All frontier models fail.
• Visual spatial reasoning: Both flawless
• Translation to european languages: Glimmer thinks very briefly, provides low error output. Qwen thinks 10 times more heavily and provides better quality tanslations.
• Small maze puzzle: Both flawless, Glimmer took 26k reasoning tokens vs Qwen 15k. Glimmers result is well explained.
• Large maze puzzle: Qwen delivery a partial solution, cheating partly. Glimmer never responded at all.
• UTF8 paraphrasing: both flawless

Agentic performance:

Here Qwen 27B appears to leave Glimmer in another league, I've not concluded my agentic tests of Glimmer 30B yet. From what I have seen Qwen codes significantly better. They do not compare.

My current results:
Glimmer is a surprisingly smart model, with a well designed architecture for local inference.
It is the first model in the sub 200B parameter class that is able to match Qwen 27B or even outclass it in some tasks.
Glimmer has a very good spatial sense
Glimmer tends to underthink where Qwen tends to overthink

For non coding tasks, Glimmer is a strong option. Faster than Qwen at similar memory footprint.
For coding tasks I'd not consider it, I'll follow up with a deeper test but from what I've seen it's not useful for most tasks.

My machine: Macbook M4 Max, 36GB.

I was using the model locally. Trying to use it to replace subscription model. I can assume local model may be performant as Claude subscription model, but the way it generated code. Changed token limit to 64K, I keep seeing the message that it was hitting token limit easily but in terminal token usage percentage was almost always 48%. The funny thing is I had to instruct few times it to do a dummy landing page with plain html. But when I tried to convert it to a react app, just the landing page, I could not complete. It keep doing incomplete response, no error was shown.

What can I do to improve this?