TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.1 to 21.9 tok/s. The julianmb/q38rocm FP4 fork gives 26.7 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, wall-clock tok/s from the API usage numbers, best of 2 reps, on AC power. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt):

Build and model	MTP off	MTP on	Draft acceptance
ROCm + Q8_0 (29 GB)	7.1	21.9 (draft 5)	73%
Vulkan + Q8_0 (29 GB)	7.4	21.3 (draft 5)	73%
ROCmFP4 FAST (14.6 GB)	12.5	26.7 (draft 6)	86%

Draft length sweep, code prompt tok/s:

draft n	2	3	4	5	6	7	8
Q8_0	15.4	18.4	19.1	21.9	21.2	21.8	20.5
FP4	-	23.3	24.9	25.0	26.7	25.6	-

Observations:

• MTP really makes a difference 2.5x to 3.1x on Q8_0, 2.1x on ROCmFP4, at 73-95% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
• Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4.
• FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 26.7 tok/s with MTP, which is 3.8x my stock Q8_0 baseline.
• Engine choice barely matters for decode at this size. ROCm, Vulkan and the rocWMMA build all land within about 3%. Vulkan does lose roughly half its prompt processing throughput though, so ROCm stays my default.
• Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
• One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-158 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag was used to ensure the GPU layers were fully utilized.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.

Flag	Value	Set by
--host	127.0.0.1	LlamaStash (loopback policy)
--port	41100	LlamaStash (auto-assigned)
-m	<model>.gguf absolute path	LlamaStash (resolves the name ref)
--mmproj	mmproj-F16.gguf	LlamaStash (auto-paired sidecar; Q8_0 rows only, FP4 has none)
--jinja	on	LlamaStash (chat template handling)
--spec-type	draft-mtp	LlamaStash (translated from --mtp on)
--spec-draft-n-max	5 (Q8_0) / 6 (FP4 best) / omitted on default rows	LlamaStash (--mtp-draft-n); llama.cpp default 3 when omitted
--fit-ctx	16384	LlamaStash (ctx knob = auto, fit hint)
--cache-type-k / -v	f16 (q8_0 on kvq8 variant rows)	llama.cpp default except for kvq8 runs
--flash-attn	on (off on the fa-off variant row only)	llama.cpp default except for fa-off runs
--n_ctx (final)	262144 per slot (native 256K)	llama.cpp fitter
--n_parallel	4, kv_unified true	llama.cpp (auto)
--n_gpu_layers	all (-1)	llama.cpp fitter (llamastash knob was auto)
--cache-type-k/v (final)	f16 / f16	llama.cpp default
--n_threads	16 (of 32 logical)	llama.cpp default
--n_batch / --n_ubatch	server defaults, not overridden	llama.cpp
spec internals (fork only)	n_min=0, p_min=0.75, backend_sampling=1	llama.cpp (q38rocm fork defaults)
draft context (fork only)	all GPU layers, f16 KV	llama.cpp (fork)

I'm the developer, and this is my project, so read it with that in mind.

The part I think is actually interesting is a refusal, so I'll start there. My app can run AI-generated shell commands and Python inside a Windows sandbox using AppContainer. Before it treats a sandbox level as available, it tests on your machine that Windows is actually enforcing that boundary, and it fails closed when a sandbox level you chose can't be established: the command is refused rather than run with weaker protection. The UI always shows which isolation level actually ran, because "sandboxed" is a claim and the app should have to prove it.

The app is Nyx AI, a Windows desktop workspace built around an AI agent. It does documents, spreadsheets with live formulas, slide decks, PDFs, web research and data analysis as well as code editing and shell work. You bring your own model: Ollama running on your PC with no API key stored in the app, or your own OpenAI, Anthropic or Ollama Cloud account. It ships no models and no inference runtime, so local means installing Ollama yourself first.

Honest qualifier on "local-first", because you'd catch it anyway: a local Ollama model keeps ordinary prompts on your PC, and small background helper calls can use a local model when one is available. On the Auto routing setting, though, heavier coding, reasoning and planning work normally prefers a cloud route you've configured. Local-first is a supported way to run it, not a guarantee that nothing ever leaves the machine. The docs say exactly that.

There are four isolation levels: Automatic, Standard, Sandbox + internet, and Sandbox · no internet. They use two Windows mechanisms: AppContainer for the two Sandbox levels and a Job Object for Standard. Standard is documented as not being a filesystem or network jail.
At the no-internet level, the sandbox is created without internet capability, so the operating system itself refuses outbound traffic from AI-run code. That still isn't an air gap for the whole app: chat still goes to whichever model provider you chose, and the update check, dependency installs you approve, and one-time component downloads can still touch the network. The threat model lists all of it.
Approvals come in two kinds. Ordinary ones are cards inside the app. Higher-risk model actions and certain sensitive operations can require a native Windows confirmation dialog outside the model-controlled interface. Nothing sent from the chat window can mark an action pre-approved, and if the dialog can't be shown the action is refused.

A smaller set of prohibited actions is refused outright in every mode and never offered for approval at all.
There's also an audit trail: a plaintext action log, plus an encrypted, hash-chained forensic log of approvals, denials and sandbox blocks. If a recorded entry is later altered, verification can show it. The docs also state plainly that this cannot prove nothing is missing.
Now the parts you'd find out anyway, so here they are first.
Windows 10/11 x64 only, because the sandbox is built on Windows-specific isolation. There is no macOS, Linux or Windows-on-ARM build.
It is proprietary and closed-source, free for personal non-commercial use, with a separate licence required for commercial use.

Because the source is not public, I have tried to make the externally verifiable parts unusually transparent. The full threat model is public, including trust boundaries, network paths and residual risks, and you can independently monitor its network behaviour.
It's beta software. It has not been independently audited. Prompt injection is documented as not solved, and the app itself ships the line "no sandbox is 100%".
Nyx AI sends no usage analytics, telemetry or crash reports of its own. Automatic update checks are on by default and can be disabled. A check sends the installed version, and the connection necessarily exposes the requesting IP to the update host. Installing from the Microsoft Store also means Microsoft's own platform diagnostics apply under Microsoft's terms, not the app's.
Site and threat model:
https://nyxai.uk

Install is through the Microsoft Store. The website does not host the installer.

Two things I'd genuinely like opinions on.
Where would you draw the line between approvals that deserve a hard OS dialog and ones where an in-app card is enough?
And if you read the threat model, what's missing from it?

https://www.youtube.com/watch?v=-UbYdRgwRWE

I've been building CortexAgent as a personal project for a while, and I finally got it to a place I'm happy to share. It's a private, local AI coding agent — everything runs on your own machine via llama.cpp. No cloud, no account, no API key, and no data ever leaves your computer. It binds to 127.0.0.1 only.

It's MIT licensed at https://github.com/greyok00/cortexagent.

One terminal interface, one local stack — the only interface you talk to is a single clean TUI (cortex). A 35B MoE does the heavy lifting on your GPU (16 GB+ VRAM recommended), with a small "overseer" model for planning and routing. Nothing else to juggle.

The processing pipeline you can watch — every request animates live through its stages, with real percentages (it never fakes a determinate number):

- 🧠 preparing — organizes your request and checks the context it needs.

- 🗜️ slimtoken — minifies your context before it reaches the model, so you fit more into the window.

- 📤 sending — the request goes to the big model.

- ✨ generating — streamed output builds in place.

- 🔧 tool wait — when the model calls a tool mid-request.

Overseer routing — a dedicated small model plans and routes each request to the big model, instead of one model trying to do everything.

Memory that actually remembers — hot working memory for the current session plus curated cold knowledge, distilled automatically from what you've already said, so you don't re-explain yourself across sessions.

Domain memory — recalled context from your own notes is injected automatically when it's relevant, no prompting needed.

Talk instead of type — speech-to-text as a floating popout you control with the mouse and your voice only (no keyboard), transcribed locally with faster-whisper. Open it from the system tray under STT Controls.

Tools & integrations

- 🌐 browser control — drive Brave over Chrome DevTools Protocol, exposed as 9 brave_* tools registered directly in the tool registry.

- 🔌 MCP client + lazy proxy — talk to MCP servers, spawning the real server only on first use instead of holding every socket open.

- 🔎 Firecrawl — web research and page parsing routed through a proxy.

- 🎨 image & video generation — SD 1.5 / SDXL / LTX run in-process through HuggingFace diffusers on the GPU the daemon already manages — no separate GUI server.

- 📄 PDF & document knowledge — ingest PDFs and documents into a queryable knowledge base.

See the whole thing — a web UI with a three.js 3D cortex scene (gold neural cortex + glass chat), a system tray dashboard, a session bridge so the TUI, web UI, and overseer all agree on context, and a live status ticker.

Under the hood

- ⚙️ Daemon lifecycle — the model + proxy run as an always-on service (systemd); VRAM-aware fallback swaps models when GPU memory gets tight.

- 📅 Scheduler + worker pool + heartbeat — background jobs run and recover cleanly.

- 🛡️ Reliability layer — pre-flight gate, anti-hallucination + post-response verification, a doctor that repairs settings drift, loop guard, chain diagnostics, and observability.

- 🛤️ Pathway capture — records each run's processing chain (frame → domain check → route → generate) as replayable runs.

- 💾 Snapshot / backup scripts — easy state backup.

Privacy isn't a feature, it's the baseline — everything binds to 127.0.0.1, and there's no cloud account to leak to. The honest caveat: the processing animation shows real percentages only; if a stage is a wait, it says so instead of pretending to progress.

What I'm asking for:

- Recommendations — what would make this a daily driver for you? Better docs, packaging/installers (pip/Homebrew/AUR), more model backends, UI polish, more tools?

- Contributors — it's a solo project; help with docs, testing, packaging, and hardening is very welcome. Issues, PRs, and "your docs confused me here" all help. Even a quick honest "here's where it breaks" is hugely useful. Thanks for reading! 🙌

Hi everyone,

I’d like to share an open-source systems project I’ve been working on called Olivia (named after my cat!).

It’s an enterprise-grade infrastructure harness written from scratch in Rust, specifically designed to run agentic LLM workflows safely. The core philosophy is to execute AI-driven actions and tools within strictly sandboxed environments using WebAssembly (Wasm/WIT).

I wanted a robust, native infrastructure to handle LLM agent workflows without compromising on security or relying on bloated software layers. Olivia ensures that the agent's interactions with databases or external scripts happen within a secure, controlled boundary.

You can check out the repository here: https://github.com/helloIAmPau/olivia

I would love to get some feedback from this community on the architecture, or hear what other sandboxed tools you'd find useful. Contributions, code reviews, and suggestions are more than welcome.

I’ve been hacking on Picot, a desktop app for using Pi coding agents.

https://github.com/shixin-guo/picot

When I first built it, it was mostly just a nicer way to chat with an agent inside a project. Useful, but still pretty rough.

Over the last month I’ve been adding the things I kept wishing existed while actually using it:

• a Skills page, so I can browse/install/enable skills from the app
• a cleaner settings page for models and providers
• thinking-level controls
• Terminal panel
• a Git panel for checking workspace changes
• better file support, including file icons, previews, resizable panels, and files mentions by @
• better session history, titles, and remembered state
• a clearer live status indicator so I can tell when the agent is still working
• general UI cleanup so it feels less like a prototype

The main difference from a month ago is that Picot no longer feels like just “chat with an agent.” It’s starting to feel more like a small desktop workspace for agent-heavy coding: sessions, files, Git, skills, settings, and status all in one place.

I still use the terminal a lot, so I’m not trying to replace it completely. But when I have multiple projects or long-running agent sessions, having a visual place to keep track of everything has been surprisingly nice.

I built Munder Difflin, a free and open source desktop app (Electron) that runs a multi-agent harness locally on your machine. It wraps CLI agents you already have, Claude Code included, so your clone works with your actual setup instead of making you migrate anything. It supports 10 CLI agent providers off the shelf.

What it does: you give it your workflows and it runs them on your machine while you’re away. Talk mode lets you orchestrate by voice, MemPalace gives agents a shared memory so they hand off work to each other, and Slack and webhook triggers let you kick off runs remotely.

How Claude Code helped: I built basically the whole thing with it. 300+ commits on the repo are co-authored by Claude. It did the Electron scaffolding, most of the agent orchestration layer, the process management for keeping long-running agents alive, and a lot of the refactors when I went from supporting one CLI provider to ten. The voice orchestration in Talk mode was almost entirely Claude Code sessions.

It’s free. Read the repo or grab the build from GitHub, no account, no key required beyond whatever CLI agent subscription you already have. I’ll add repo in comments

Happy to answer questions about the architecture.

Yeah, another qwen post… I love the new model but the speed on a spark even on fp8 is abysmal. I am getting about 15-17tok/s tops with mtp=2.

I saw there is a dspark drafter trained but it requires only sglang and am getting compatibility errors

Trying this drafter: https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark

So wondering if anyone has luck, I was very impressed with Muse Glimmer’s drafter was wondering if I can get an performance benefit from using a diffusion drafter here as well.

If there is a good dgx spark sglang image to use please do share!

Thanks!

I used the pcie 4x and one of the M.2 slots with respective raisers and external PSU.
Idea was to check: how far can I bring this platform, the 128GB vram is great but iGPU is kinda slow and bigger models struggle on it.
Also DSV4 flash sounds really nice but runs quite poorly on Strix Halo alone.

Here are some models I ran, as my workflows are mainly 'low concurrency' the tests were aimed at measuring single stream inference, not concurrent inference.

Please let me know if you would do anything different / try any other interesting model!

*Edit(i) The UD Q8 results of DS4 Flash (Q8_K_XL) are also computed now and its very close to Q4: pf 236 / tg 19.5

Last month I posted about wiring the G2 into the AI coding agents already sitting on my Mac, so the glasses answer like they know what I do all day instead of like a search box. That post went way better than I expected, and honestly the DMs have been the best part of it. (Original post here if you want the architecture and the SDK gotchas.)

Three weeks later it's basically different software, and about 250 people have set it up so far. Here's where it's at. The giveaway details are near the bottom.

GotCOS has a Mac app now

https://preview.redd.it/lckrpa72mdjh1.png?width=3000&format=png&auto=webp&s=036926f0eea43240c8a91d7924e4aba7a364c6f5

Version one was a terminal command and a lot of faith. Now there's COS Control, a menu bar app that runs the server for you, shows you what's actually running, updates without breaking itself, and can repair common install or update failures. After setup, you shouldn't need the terminal for normal use.

It opens on an Activity view, which I did not expect to use nearly as much as I do. Six windows into what your COS is already holding: Messages, Speakers, Meetings, Memories, Threads, and your agent Sessions. You stop guessing what it caught. Every meeting, every answer, every voice it's learned is sitting there in a list you can open.

It knows who talked

https://preview.redd.it/jmdhc872mdjh1.png?width=3200&format=png&auto=webp&s=84b2813ff735406f1c9d67b1bd1b79ffaaa3e8e6

This one surprised me. Meetings get transcribed locally, and now voices carry across them, so somebody you talked to three weeks ago shows up by name in today's call instead of as "Speaker 1" all over again.

It was hard, and I built the review tools before I trusted the accuracy at all. Below a confidence floor the panel just says "unidentified" rather than guessing, because a confident wrong name is worse than no name. You can play the audio back, confirm a voice, or tell it "nope, wrong person" and it'll retract whatever training that mistake created. With a microphone strapped to your face in a real room, that correction loop is doing most of the work.

Everything else that shipped

https://preview.redd.it/hjiqq772mdjh1.png?width=3200&format=png&auto=webp&s=afd31d109162e34168db1630855fae697282a1f7

• Meetings show up faster. A saved meeting used to wait on the full high-quality pass before it landed anywhere. Now it appears right away, then the higher-quality pass fills in the rest.
• Transcription got faster when nothing else is going on. The post-meeting polish grabs the GPU when nothing live is competing for it, and hands it back the second a meeting starts. Roughly 6x faster per word than the CPU path.
• Live text on the lens while you're still talking, swapped for the real transcript once it commits.
• Video uploads resume now instead of dying on a bad connection.
• Photos on the lens at 288x144, plus answer images you can pull up on the G2.
• Jobs still live on the Mac, so you can close Even Hub or drop off Wi-Fi and the answer is waiting when you get back.

Cursor works now, and there's no second bill

Not just Claude Code and Codex anymore. Cursor works too, so you can put Grok or Composer behind the glasses. You pick the model per question from the lens and swap mid-conversation if one isn't getting it. Three agent CLIs, whichever one you're already paying for.

Which gets to the thing people keep assuming wrong. GotCOS doesn't add a separate API bill. It signs in through the CLI you already use, so it runs on the Claude Code, Codex, or Cursor account you already have, subject to that provider's plan and limits. Nothing to paste, nothing metering you per token.

Whatever can run entirely on your machine already does. Transcription is local by default through whisper.cpp, so meeting audio never leaves the Mac, and spoken replies can be local too. Cloud transcription is there as a fallback but it's off until you turn it on, and just configuring a key won't upload anything on its own. To be precise, since this crowd will ask: the reasoning still goes to whichever provider you signed into, on your existing plan. What's local is the transcription, the speech, and the files. Meetings, memories, and transcripts sit on your disk as plain files you can open, grep, back up, or delete.

It works inside the Hub, not around it

Saying this plainly since there are a few different approaches floating around this sub. It's a marketplace app. You install it from the Even Hub like anything else. Nothing to flash, nothing to unlock, no warranty to worry about. I build inside the Even Hub SDK sandbox and code around whatever it hands me.

That's a tradeoff and I'm not claiming it's the better one. The custom firmware folks get past walls I have to engineer around, and I've picked things up from their work. But if you want your glasses doing more without modifying the glasses, this is the path.

What you end up with is a workflow that stays on hardware you already own. Ask a question, read it on the lens. Talk, and watch the transcript appear while you're still talking. Point the server at a folder and the agent works through the files in it, including whatever your Mac already syncs down from iCloud Drive. A meeting you just recorded becomes a file the agent can read a minute later, so you can ask about it without exporting anything anywhere.

Two people this owes something to

u/ICE_MF_Mike ran my server through his own Claude setup after the last post and came back with an actual security review. Shell injection in a title generator, file permissions, network defaults, a Telegram switch that should have been opt-in. He was right about all of it, and all of it is patched and shipped. Best thing that's happened to this project.

u/True-Benefit-4856 is the only reason there's a desktop app. The companion started as their idea and turned out to be the one change that made this usable for anyone who doesn't want to live in a terminal.

If you read the source and find something, I want to hear it. It's MIT, it's on npm, and it runs on your machine, so read it before you run it.

The giveaway: a pair of G2s

https://www.gotcos.com/challenge/

I've got a second pair sitting here and I'd rather they go to somebody who'll do something interesting with them than sit in a drawer.

To enter: comment with what you'd build or how you'd use it. Not a wishlist. The actual use case, the workflow, the thing you can't do today and want to. Feature ideas for COS Glasses count, and so does a completely different application of the hardware.

How I'll pick: I read all of them and pick the one I most want to see exist. Best idea wins, upvotes don't decide it, though I'll definitely read the ones you upvote.

Deadline: October 1.

Fine print: used pair, good condition, mine to give. Open worldwide, and the winner covers shipping from the US plus any customs or duties on their end. I'd rather send these to the right person in another country than limit it to one. You don't need to be running COS Glasses to enter, and running it doesn't help your odds. If your idea wins and it's something I can build, I'd rather build it with you than for you.

Setup

• Setup wizard, about five minutes: https://www.gotcos.com/wizard/
• The Mac app: https://www.gotcos.com/control/
• Full docs, every feature on one page: https://www.gotcos.com/docs/
• Source, MIT, on npm: https://www.npmjs.com/package/@gotcos/glasses-server
• Even Hub: search “COS Glasses”

You need at least one of Claude Code, Codex CLI, or Cursor installed and signed in. Whichever you already have is fine. It works standalone, and if you're already running a bigger COS setup, point the server at that folder and the glasses inherit all of it.

I keep calling this a side project and it keeps being the thing I use every single day. Recording a customer call and asking what I missed. Logging an idea halfway through a walk without pulling my phone out.

If you try it and it breaks, tell me and I'll get you running. Building against this SDK yourself? Happy to compare notes on any of the traps. And if you've got an idea for what this should do next, drop it in the comments, because there's a pair of glasses in it for one of you.

A coding agent called Zero launched today and the pitch caught my attention: your model, your machine, your rules.

It's built from scratch in Go — single binary, no Python environment, no dependency hell. Install is one line:

npm install -g u/gitlawb/zero

Repo: https://github.com/Gitlawb/zero

What stood out vs the usual coding agents:

- You pick the model provider, it's not locked to one API

- Runs on your machine, not their cloud

- Go means it's lean — the team behind it (GitLawb) claims 5x faster than their previous harness

It's early ,someone already flagged that only one provider can be active at a time, and the maintainers said to file a feature request. So rough edges exist but they're responsive.

Anyone tried pointing it at a local model through Ollama yet? Curious how it handles smaller coding models vs the big API ones.