u/Temporary-Owl1725

We run HexGrid Cloud, a platform for deploying open-source models on GPUs, and we're heads-down optimizing our serving/deployment layer.

To pressure-test it we're benchmarking real models under real concurrency — and instead of guessing, we'd rather run what you actually want to see.

---

Models available for benchmarking:

• Nemotron-3 Super 120B-A12B (only NVFP4)
• Nemotron-3 Nano 30B A3B
• Qwen-3.6 27B
• Llama 3.3 70B Instruct
• Gemma-4 31B
• Devstral-Small-2-24B-Instruct-2512
• ?? (you suggest a model to us)

We're focused on chat/instruct models for now (that's what most of our users deploy), so pick one from the list above — or suggest another open-weight chat model that fits on a single H200 (141GB).

---

Hardware & quant choices:

• GPU (up to H200 for this round): RTX PRO 6000 · L40S · H100 · H200
• Quant: FP8 / AWQ / BF16
• Context length: (8K, 32K, 64K, 128K)
• What you want measured: max throughput? single-stream speed? long-context prefill?

---

We'll run the top picks and post full results — tokens/sec, TTFT, TPOT, throughput under concurrency, and cost-per-million-tokens — config and flags included so it's reproducible.

Let us know in comments.

We run HexGrid Cloud, a platform for deploying open-source models on GPUs, and we're heads-down optimizing our serving/deployment layer.

To pressure-test it we're benchmarking real models under real concurrency — and instead of guessing, we'd rather run what you actually want to see.

---

Models available for benchmarking:

• Nemotron-3 Super 120B-A12B (only NVFP4)
• Nemotron-3 Nano 30B A3B
• Qwen-3.6 27B
• Llama 3.3 70B Instruct
• Gemma-4 31B
• Devstral-Small-2-24B-Instruct-2512
• ?? (you suggest a model to us)

We're focused on chat/instruct models for now (that's what most of our users deploy), so pick one from the list above — or suggest another open-weight chat model that fits on a single H200 (141GB).

---

Hardware & quant choices:

• GPU (up to H200 for this round): RTX PRO 6000 · L40S · H100 · H200
• Quant: FP8 / AWQ / BF16
• Context length: (8K, 32K, 64K, 128K)
• What you want measured: max throughput? single-stream speed? long-context prefill?

---

We'll run the top picks and post full results — tokens/sec, TTFT, TPOT, throughput under concurrency, and cost-per-million-tokens — config and flags included so it's reproducible.

Let us know in comments.

Google reportedly restricted Meta’s access to Gemini models because demand exceeded available compute capacity. The report says Meta pushed employees to use AI tokens more efficiently after internal projects were affected.

Italy-based Domyn says it plans to launch a fully open-source frontier model within a year, targeting more than 400B parameters. The project is part of Europe’s push to reduce dependence on foreign-hosted AI systems and is tied to the EUROPA consortium and EuroHPC infrastructure.

We benchmarked Qwen-3.5 9B BF16 on our custom bench on RTX 5090 [1-GPU] using real world ShareGPT dataset.

TL;DR:

Found a clean ceiling: throughput climbs nicely up to concurrency 64 (~1280 tok/s output) and then just... stops. 128 concurrency level gives basically the same throughput but nearly doubles end-to-end latency and triples time-to-first-token (5.7s → 17.9s p95).

So past 64 concurrency , GPU is not getting more work done — it's just making requests wait longer in the queue.

Details:

Model

• Model: Qwen/Qwen3.5-9B
• HF Path: Qwen/Qwen3.5-9B
• Quantization / dtype: BF-16
• Context length configured: 4096 max-tokens

Serving

• Engine: vllm 0.19
• Cuda: 13.0.1
• Engine flags:{'enable_auto_tool_choice': True, 'exclude_tools_when_tool_choice_none': True, 'tool_call_parser': 'qwen3_coder', 'dtype': 'bfloat16', 'max_model_len': 4098, 'served_model_name': ['Qwen/Qwen3.5-9B'], 'generation_config': 'vllm', 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'language_model_only': True, 'max_num_batched_tokens': 4096, 'enable_chunked_prefill': True}
• Endpoint: /v1/chat/completions

Hardware

• GPU: 1x RTX 5090
• VRAM: 32GB
• CPU: 48 vCPU   |  177 GB RAM

Workload

• Dataset: ShareGPT sample, [1080 unique prompts] x [4-concurrency settings] => Total 4320 prompts
• Conversation shape: Multi-turn response per request
• Languages: Multilingual with en/zh/ru/th/ko/fr/pl/ja
• max_model_len: 4098
• max_tokens per completion: 256
• temperature: 0.2

Methodology

• Load tool: Custom Harness (currently building but will be public soon)
• Concurrency Request levels: 16, 32, 64, 128
• Streaming: ON

Metrics

Some charts:

Benchmark started at <<09:41&gt;> in the charts and stopped at <<10.01>>. Benchmark was run first for 16 concurrency, then 32, 64, 128 and the performance flattened out after 64.

https://preview.redd.it/59ln7nq3uk7h1.png?width=2270&format=png&auto=webp&s=8714a3db41aea0f3543aa288d1bcf84a779c7238

https://preview.redd.it/gywpnk37uk7h1.png?width=2260&format=png&auto=webp&s=27330645966441cb8543be834c002c769963a621

https://preview.redd.it/koq003lvuk7h1.png?width=2296&format=png&auto=webp&s=cae66423d0d361bf675cca866849488c61558431

Anybody here was able to achieve a higher output for this and can please constructively criticise our deployment run?

Hi folks,

We're a small team of Engineers (experience ranging from 8-15 yrs) in FAANG & other Big Tech companies and are building an AI startup in the Agentic AI space. We're in the final discussions of raising a small seed round. And now are looking for Interns and Founding engineers to join us.

For Interns: We can pay a stipend b/w Rs. 5k to 15k per month, depending on the skillset they bring to table. No equity.

If you're interested, please DM me your LinkedIn profile. If we like it we'll send you more info about the company and your role and will initiate more discussions.

Best,

Hi folks,

We're a small team of Engineers (experience ranging from 8-15 yrs) in FAANG & other Big Tech companies and are building an AI startup in the Agentic AI space. We're in the final discussions of raising a small seed round. And now are looking for Interns and Founding engineers to join us.

For Interns: We can pay a stipend b/w Rs. 5k to 15k per month, depending on the skillset they bring to table. No equity.

For Founding Engineers: We're looking to go only for the Equity route right now, depending on the skillset. No salary as of now but it should change soon. Min. requirements are 3+ yrs of Backend experience in any language/stack.

If you're interested, please DM me your LinkedIn profile. If we like it we'll send you more info about the company and your role and will initiate more discussions.

Best,