u/StartupTim

Hello,

I will be picking up 4x AMD 495+ systems with 192GB of unified memory and I am looking to cluster them in a way similar to doing tp=4?

What's the best way to do this?

Would clustering over the 10GbE work? I think that would be too slow. How about over the USB 4 80Gbps, there a way to utilize that as super low latency? I think it has 4x USB 4.1 ports, so perhaps a cross connect, ot if not, a daisy chain token-like?

Any ideas on the best route to cluster 4x?

Thanks

I just want an Android watch that lets me add my own programs to, custom APK loading.

Any idea what watch I could get for this? I just want it to be a big watch, big battery, and load my own APKs.

Thanks for your suggestions!

I need to buy 2x DGX Spark but can't find a seller. Any hope or idea where I can get 2?

The price seems to have soared. Local Microcenter had 25+ then 0 the next day. They pulling stock?

I love the work that Unsloth does with all their GGUFs. It is truly remarkable.

But I can't figure out a reliable way, or really any way, to shard or otherwise TP=2 the model across multiple physical systems (2x Nvidia DGX Sparks 128GB unified memory). This would effectively allow us to split model weights across multiple nodes while also increasing speeds.

If this worked then all of us DGX Spark owners could run a ton more models IMO. vLLM with tp=2 works great but doesn't really support multi-file GGUFs.

Is there a way to do this or can we get some help, maybe Unsloth creates a vLLM fork that is exclusive to GGUFs?

Thanks

Hey all,

I have a Windows VM that I want to backup via PBS, but not just the image, hopefully interior files and such.

Is there a good way to do this? Any thoughts or tips?

Thanks in advance all!

I'm trying to get a model to reverse engineer / decompile or otherwise reverse to source some of my old c,c++, pascal, and asm demo programs I made from decades ago and I'm constantly met with refusals. It is highly annoying.

Does anybody know of a good abliterated/uncensored version of Qwen3.8 27b that reliably will stop refusing such basic things?

Thanks!

Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an issue where it locks up when RAM fills, so having more headroom allows things to be comfortably more stable.

Question: DeepSeek-V4-Flash-0731 (304B MoE) at full 1M context on 2x NVIDIA DGX Spark — how can we free more RAM for the OS?

Hardware

• 2x DGX Spark (GB10 Grace Blackwell, SM121): 128 GB unified memory each (~121.7 GiB visible), ~273 GB/s bandwidth, 20-core ARM, 4 TB NVMe, DGX OS (headless, GUI/desktop services disabled)
• Direct-connected ConnectX-7 fabric, dual links, RoCE, MTU 9000; NCCL confirmed using IB transport (~200 Gbps)

Model

• DeepSeek-V4-Flash-0731, native FP8 checkpoint (167 GB safetensors)
• 304B total params, MoE: 256 routed experts/layer, 6 active + 1 shared per token, 43 layers
• Sparse attention (indexer top-k 512) + built-in KV compression (4x / 128x per-layer groups), YaRN-calibrated 1M context

Serving stack

• vLLM 0.26.1rc1 built from source (the DeepSeek V4 SM12x support PR), CUDA 13.0 toolchain, running in containers
• Tensor parallel TP=2 across the two boxes (--nnodes 2, multiprocessing backend, no Ray)
• DSpark speculative decoding (method "dspark", num_speculative_tokens=5) — ~80% draft acceptance
• llama-swap as the single OpenAI-compatible endpoint / model router; Open WebUI frontend

Key engine flags

• --kv-cache-dtype fp8_ds_mla (8-bit packed 584 B/token/layer; measured ~3.9 KB/token effective across all layers with the model's compression)
• --max-model-len 1048576 (full 1M)
• --gpu-memory-utilization 0.82 ← the knob in question
• --max-num-batched-tokens 4096 (larger values blow up per-request block accounting via tiny-block-size compressor-state cache groups — each 4-token block pins a ~1 MB slab)
• --max-num-seqs 6, prefix caching + chunked prefill on, CUDA graphs on
• tokenizer/tool/reasoning parsers: deepseek_v4

Memory anatomy per node (measured)

• Weights + activations + CUDA graphs: ~87–89 GiB (immovable; weights are ~83.5 GiB/node under TP=2)
• KV cache pool: ~11.2 GiB → ~1.6M tokens capacity (1.53x concurrency at 1M)
• Left for OS + services: ~5–7 GB idle, dips to ~2–3 GB during marathon 1M-context sessions (slow host-side leak, ~100–200 MB/h under sustained long-context load)
• Watchdogs: earlyoom at absolute 1.5/1.0 GiB floors, custom graceful unloader at 2 GB, vm.min_free_kbytes=2 GiB, swap OFF (on unified memory, swap turns a clean OOM kill into a 30-min livelock)

Performance (so you know what we don't want to lose)

• 82 tok/s decode single-stream, ~1,400 tok/s prefill, 146K-token needle test passes 3/3

What we've already ruled out (please don't suggest these)

• Lowering gpu-memory-utilization below 0.82: at 1M the KV pool hits vLLM's serve-one-request floor and it refuses to start
• Lowering max-model-len alone: frees nothing — the pool is sized by the budget %, not by context
• cgroup/docker memory caps on the model containers: reclaim-loops on file-backed pages and starves sshd (tried, worse than nothing)
• Enabling swap: livelocks the whole box on unified memory
• CPU offload / --cpu-offload-gb: meaningless on unified memory (one pool)
• LMCache disk tier: incompatible — asserts on the packed fp8_ds_mla layout ("kv_cache last dim must be 584, got 512")
• Weight streaming from NVMe: 40x bandwidth gap, measured ~14 tok/s vs 82

The actual question

On identical unified-memory boxes, what host-side or vLLM-side levers actually reduce resident overhead? Specifically curious about:

• Shrinking vLLM's host-process footprint (API server + engine + worker processes RSS)
• NCCL buffer/registration tuning for 2-node TP (NCCL_BUFFSIZE etc.) — free GBs or just MBs?
• torch.compile / inductor / CUDA graph memory: is --enforce-eager's memory savings worth the latency hit on Blackwell, and has anyone measured it for big MoE?
• Anyone found the source of slow host-RSS growth (~100–200 MB/h) in long-context vLLM serving?
• Any DGX OS / GB10-specific reservations (driver, carveouts) that can be trimmed safely?

A huge thanks in advance for any and all your responses!!

Hey all, I'm getting a bunch of errors when using DS4Flash 0731 with the app_diff tool call and can't seem to figure out what to do. Could you peeps smarter than me offer some advice?

Setup

• 2× NVIDIA DGX Spark (GB10, SM121, 128 GB unified each), tensor-parallel TP=2 over a ConnectX-7 RoCE fabric
• deepseek-ai/DeepSeek-V4-Flash-0731, native FP8, full 1M context, KV nvfp4_ds_mla
• vLLM 0.21.1rc1 (DSpark recipe build), --tool-call-parser deepseek_v4, --reasoning-parser deepseek_v4, --enable-auto-tool-choice, DSpark spec-decode k=5
• llama-swap as an OpenAI-compatible front end on :8000
• Harness tried various such as Zoo/Roo Code 3.67 in VS Code, OpenAI-compatible provider

The problem

Zoo/Roo Code intermittently corrupts source files. The cause is that the model emits apply_diff tool-call arguments where the diff string contains one or more <<<<<<< SEARCH blocks but no >>>>>>> REPLACE terminator at all. Occasionally it also omits the required path parameter entirely, and Roo reports:

Roo tried to use apply_diff without value for required parameter 'path'. Retrying...

Roo applies the half-formed diff, which duplicates functions and cascades into more corruption. The model then correctly notices the file is broken and rewrites it — so it's recoverable, just destructive and slow.

What I've ruled out (with measurements)

Suspect	Result
Proxy rewriting the request/response	llama-swap only rewrites the model field. Tested direct to vLLM — same behavior.
Response truncation	Tool args of 795 / 887 / 2006 chars all arrived as valid JSON with path and both markers — non-streaming and streaming.
Temperature	4/4 well-formed at temp 1.0 and 0.3.
Reasoning disabled	The recipe shipped thinking:false. I enabled reasoning_effort=max + top_p=0.95 per the model card. Still occurs.
Known cold-prefill garble patch	Patch 3 and 4 is loaded (grep -c is_prefill_chunk scheduler.py = 5 on both nodes).

The confusing part: it's not reproducible on demand.

One run of a multi-block diff prompt failed 0/3 — every trial produced SEARCH=2, REPLACE=0, i.e. two SEARCH markers and zero REPLACE markers, even though the prompt explicitly demanded all three markers. A near-identical rerun minutes later passed 4/4 with correctly balanced blocks. Single-block diffs have passed 4/4 every time I've tried.

Questions

1. Is this a known weakness of the deepseek_v4 tool-call parser in vLLM, or of DeepSeek-V4 itself with SEARCH/REPLACE-style diff tools?
2. Is anyone running DeepSeek-V4-Flash with Roo Code / Cline / Kilo successfully for agentic coding? What settings?
3. Could speculative decoding (DSpark k=5) plausibly drop tokens mid-tool-call in a way that loses a terminator without invalidating the JSON? I haven't found a way to test this cleanly.
4. Roo removed the XML tool-protocol selector in v3.37 (PR #10281) and closed the request to restore it as "not planned" — so I can't fall back to XML tools. Any other way to force prompt-based tool calling with an OpenAI-compatible backend?

Bonus observation (may be related)

Enabling tools at all destroys streaming granularity. Measured on the same prompt:

• without tools: 117 SSE deltas, avg 12.9 chars, max 42, longest gap 0.45 s
• with tools: 2 deltas, avg 417 chars, max 833, longest gap 5.68 s

So the tool-call parser buffers arguments rather than streaming them incrementally. Is that inherent to vLLM's tool parsers, or specific to deepseek_v4?

Any ideas on what I could do to fix this? I can test whatever you guys have and give you feedback.

Many thanks in advance!!

Hello,

I am struggling to find the best model and workflow that can take multiple references images, as well as reference consistency images, and then a text prompt to create new images.

Flux for example, when I give it reference images, it doesnt seem to merge it into the final page very well, and consistency references fail entirely.

Please and thanks for your suggestions!

Hey all,

So I tend to favor the Claude Desktop app in Code mode as the GUI does a great job of previewing code, MCP browser interactions/screenshot evals/etc.

But I recall people saying they could get Claude Desktop to use a local API. Does anybody know how to do this? Specifically, where can I configure Claude Desktop to point to one of my local llm APIs. I don't need help setting up ollama/llama.cpp/vllm etc.

Thanks

I have a system with dual GPUs and 64GB of ram and I'm looking for the best model and LLM software to do agentic coding (largest context possible)/tool use/hermes.

Which model and llm software would you recommend?

Thanks

Hey all,

So I have a Nvidia DGX Spark and an AMD Strix 395, both have 128GB of unified memory. The Spark has 200Gbit network and the AMD Strix has 5Gbit ethernet (but it has a pcie gen 4x4 slot).

Is there any chance I can cluster the 2 together to run a larger model that can fit in the ~256GB (minus OS) unified memory?

I've seen that Deepseek v4 Flash can fit on 2x DGX Spark, but maybe I can use my Strix system instead?

Any ideas on if this would be possible? If so, how would you go abouts doing it? Would it help if I added a Mellanox ConnectX-6 QSFP+28 to the AMD Strix and connected it to the DGX Spark? I would have maybe 64Gbps over the 100Gbps link, but 64 is faster than 5.

Thoughts?

Thanks!

Hey all,

I have a piece of hardware laying around which is pretty fast from a traditional (non-GPU) server viewpoint. The hardware is the following:

• Dell C6525 Server with Quad Node (4x server blades) with the following:
• 2x AMD EPYC 7702 64-Core Processors
• 8 memory channels per socket so 16 channels total 512 GB of DDR4 RAM 3200MT/s
• NOTE: Math'd out, 16 channels of 3200MT/s is 409.6 GB/s total memory bandwidth
• 24x 3.84TB SATA12G SSDs (6 per server) 12GB each so pretty fast
• Zero GPU
• 4x Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (it does support RDMA)
• The above is PER server and there are four. So 2TB ram total

I've seen some videos about clustering a larger model across multiple servers for either a) Model token speed, or b) Loading larger model sizes

I think in my example, is it possible to cluster all 4 systems to run Unsloth 4bit GLM 5.2 (467GB) on each system somehow, for token speed? Or what about making 2x clusters, with each cluster loading Unsloth GLM 5.2 8bit (820GB) for both speed and larger models?

The end result is I want to load up a big model like GLM 5.2 as fast as possible on this hardware. I know it is CPU only, but the memory should hit 409GB/s per node, so it should be somewhat OK, especially if spread across 4 nodes. I just want to see the best possible with this hardware and then test it using typical agentic coding harnesses.

Any idea on how I would go abouts doing this?

HUGE thanks in advance for all your feedback/advice!

I'm using the latest Zoocode and the VSCode editor in the right frame does not show anymore and I can't get it to show at all.

Before, when Zoocode would edit a file, the editor would load the file on the right frame and you could see the file being edited in real time. You could also see the diff editing in real time.

However, now nothing loads in the editor, diffs no longer show up at all.

Also, the Zoocode setting "Auto-close files Zoo opened" is now enabled by default and cannot be disabled. If you disable it, it automatically re-enables itself.

Please help! I need to see what files are being edited, what the edit is, etc.

I am thinking of turning my Strix system into a hermes setup for automation/assistance and using 100% self-hosted stuff.

I was thinking of using Unsloths client and the Hermes desktop client together.

Anybody done this yet?

Anybody know of an MCP to speak to chatGPT using your membership to create, view, and edit images?

I'm aware and often use OpenAI's image-2 API, but ChatGPT can write great image prompts and image edit prompts when using the normal web interface.

Is there an MCP that can do this that connects to your ChatGPT subscription?

Thanks