u/styles01

Hey all, I'm trying to build a backend-connection for Stackchan to Openclaw and Hermes. Various folks have done this or tried it in different ways. If you're interested in contributing, I just started the repo but will post my progress on twitter as well:

https://github.com/styles01/StackChan-OpenClaw-Hermes

https://x.com/jaita

After 2 days of testing and benchmarking every single recipe I could find on the forums and twitter, this this the current best I have put together for single spark (i tested vllm, sglang, etc). This is tuned also for use with Hermes agent at 256k context, 5 lanes.

PLEASE POST YOURS, with your results.

https://github.com/styles01/sparkrun-recipes/blob/main/recipes/qwen-38-27b.yaml

recipe_version: "4"
name: qwen-38-27b
description: "Qwen3.8 27B NVFP4 — triton_attn, MTP k=2, fp8 KV (auto from checkpoint), 256K context, vision. Mia Lab config."
model: unsloth/Qwen3.8-27B-NVFP4
runtime: vllm
container: vllm/vllm-openai:nightly-aarch64
maintainer: styles01

metadata:
  author: styles01
  created: "2026-08-14"
  updated: "2026-08-14"
  tags:
    - qwen
    - qwen3.8
    - 27b
    - nvfp4
    - mtp
    - triton-attn
    - vision
  runbook: runbooks/qwen-38-27b.md

solo_only: false
cluster_only: false

min_nodes: 1
max_nodes: 1

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  pipeline_parallel: 1
  gpu_memory_utilization: 0.84
  max_model_len: 262144
  max_num_seqs: 4
  max_num_batched_tokens: 8192
  kv_cache_dtype: fp8
  attention_backend: triton_attn
  load_format: fastsafetensors
  tool_call_parser: qwen3_coder
  reasoning_parser: qwen3
  served_model_name: "qwen3.8-27b unsloth/Qwen3.8-27B-NVFP4"
  speculative_config: '{"method":"mtp","num_speculative_tokens":2}'
  async_scheduling: true
  enable_prefix_caching: true
  enable_chunked_prefill: true
  trust_remote_code: true
  enable_auto_tool_choice: true
  quantization: compressed-tensors

env:
  CUTE_DSL_ARCH: sm_121a

executor_config:
  entrypoint: ""
  auto_remove: false
  user: root

# NO Mamba patch needed — vLLM nightly-aarch64 has native MTP support for Qwen 3.8.
# NO enforce-eager — CUDA graphs work with triton_attn + NVFP4 checkpoint.
# The NVFP4 checkpoint auto-applies FP8 KV cache scheme (calibrated), doubling KV capacity.
# Using FP8 checkpoint (Qwen/Qwen3.8-27B-FP8) instead of NVFP4 causes OOM during MTP — that was our root crash cause.
# NOTE: max_num_batched_tokens=8192 is REQUIRED for GDN/Mamba cache alignment (32768 crashes under concurrency).
# See runbooks/qwen-38-27b.md for alternative configs (erdaltoprak GMU 0.50, SGLang FP8, Radix DSpark).

benchmark:
  framework: llama-benchy
  pp: [2048]
  tg: [128]
  depth: [0, 4096, 8192, 16384, 32768, 65535, 100000]
  concurrency: [1, 2, 5, 10]
  prefix_caching: true
  runs: 3

command: |
  vllm serve {model} \
    --host {host} \
    --port {port} \
    --served-model-name {served_model_name} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --trust-remote-code \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --enable-auto-tool-choice \
    --tool-call-parser {tool_call_parser} \
    --reasoning-parser {reasoning_parser} \
    --quantization {quantization} \
    --load-format {load_format} \
    --attention-backend {attention_backend} \
    --async-scheduling \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --speculative-config '{speculative_config}' \
    -tp {tensor_parallel} \
    -pp {pipeline_parallel}

Suno pissed the world off this week and coincidentally(?) Minimax open-weighted its excellent music generation model, that happens to perfectly fit in a DGX Spark.

https://huggingface.co/MiniMaxAI/MiniMax-Music3

https://docs.comfy.org/tutorials/audio/minimax/minimax-music-3

https://sgl-project.github.io/sglang-omni/cookbook/minimax\_music3.html

Nvidia dropped Nemotron 3.5 Lightning today: here's my recipe for a single DGX spark, 120 t/s single stream : https://github.com/styles01/sparkrun-recipes/blob/main/recipes/nemotron-3.5-lightning-30b-a3b-nvfp4.yaml

--enable-auto-tool-choicerecipe_version: "2"
name: nemotron-3.5-lightning-30b-a3b-nvfp4
description: "NVIDIA Nemotron 3.5 Lightning 30B-A3B NVFP4 — DSpark spec decode, fp8 KV, marlin MoE, mamba flashinfer, 1M ctx. From NVIDIA vLLM DGX Spark cookbook."
model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
runtime: vllm
container: vllm/vllm-openai:v0.27.1

metadata:
  description: |
    NVIDIA Nemotron 3.5 Lightning 30B-A3B NVFP4 (hybrid Mamba-2+MoE+Attention)
    with DSpark speculative decoding. Adapted from the official NVIDIA vLLM
    DGX Spark recipe (vLLM Nightly v0.27.1). 3B active MoE, ~21GiB weights incl
    draft head. TP=1 on GB10.
  maintainer: styles01
  source: "https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4"
  created: "2026-08-11"
  updated: "2026-08-11"
  tags:
    - nemotron
    - nemotron-3.5
    - lightning
    - 30b
    - a3b
    - nvfp4
    - dspark
    - mamba
    - moe
    - fp8-kv
    - marlin
    - dgx-spark

solo_only: true
cluster_only: false

min_nodes: 1
max_nodes: 1

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.91
  max_model_len: 1048576
  kv_cache_dtype: fp8
  speculative_config: '{"method":"dspark","model":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark","num_speculative_tokens":3}'
  served_model_name: "nemotron-3.5-lightning nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4"

env:
  HF_HOME: /cache/huggingface
  PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True

executor_config:
  entrypoint: ""
  auto_remove: false
  user: root

benchmark:
  framework: llama-benchy
  claimed_speed: "~124 tok/s single-stream (MiaAI-Lab DGX Spark SGLang+DSpark reference)"
  notes: |
    vLLM adaptation of the official NVIDIA DGX Spark cookbook recipe.
    Verify tool-calling (qwen3_coder) and reasoning parser (nemotron_v3)
    before agent use. sparkrun resolves {model} and the spec draft model
    from the HF cache snapshot paths.

command: |
  vllm serve {model} \
    --served-model-name {served_model_name} \
    --host {host} --port {port} \
    --trust-remote-code \
    --moe-backend marlin \
    --kv-cache-dtype {kv_cache_dtype} \
    --max-model-len {max_model_len} \
    --enable-prefix-caching \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --speculative-config '{speculative_config}' \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \