Login

u/NancyTransmed

r/LearnToReddit r/pascal r/Vllm r/isometric r/honesttransgender
▲ 1 r/Vllm

Am I the only one experiencing CUDA segfaults with Hcompany/Holo-3.1-35B-A3B-NVFP4? The model is really nice, if not for the crashes making useless.

description: vLLM serving Holo3.1-35B-A3B (computer-use VLM) in NVFP4; lean 0.28 GPU, TP=2-capable.
model: Hcompany/Holo-3.1-35B-A3B-NVFP4
container: vllm-node
defaults:
  port: 8010
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.28
  max_model_len: 131072
  max_num_batched_tokens: 16384
  max_num_seqs: 10
env:
  HF_HUB_OFFLINE: '1'
  VLLM_MARLIN_USE_ATOMIC_ADD: '1'
command: |
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
    --served-model-name holo3.1 \
    --host {host} \
    --port {port} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --kv-cache-dtype fp8 \
    --mamba_ssm_cache_dtype float32 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --limit-mm-per-prompt '{"image": 3, "video": 0}' \
    --tensor-parallel-size {tensor_parallel} \
    --distributed-executor-backend ray
recipe_version: '1'
name: Holo-3.1-35B-A3B-NVFP4
cluster_only: false
u/NancyTransmed — 1 day ago