Introduction
A note on terminology. Our previous article on Kimi-K3 used the phrase “Day 0 deployment.” A reader pointed out that when the work starts from the official public release, “Day 1 deployment” is the more accurate term. That’s a fair correction, so we’ve switched to “Day 1” from this article onward. Thanks for the feedback.
At around 1:00 AM JST on August 13, 2026, Alibaba’s Qwen team published the model weights for Qwen3.8-2.4T-A95B. This is the 2.4T-parameter flagship that was announced on August 3 as the first model in the Max class to be released with open weights.
Two weeks ago we ran a Day 1 deployment of Kimi-K3. We did the same thing here: on the day the weights went live, we deployed the model to a single node of 8x NVIDIA B300 and benchmarked inference performance. What follows are the early results. We kept the measurement conditions identical to last time, so the numbers can be compared directly against Kimi-K3.
What is Qwen3.8-2.4T-A95B?
Qwen3.8-2.4T-A95B is a frontier-class Mixture of Experts (MoE) model from the Alibaba Qwen team. The main specifications:
| Item | Specification |
|---|---|
| Total parameters | 2.4T |
| Active parameters | 95B |
| Architecture | MoE based on the Qwen3.5 architecture |
| Attention | Hybrid: Gated DeltaNet (linear attention, ~75% of layers) + GQA (~25%) |
| Precision of released weights | BF16 / FP8 |
| Context length | 256K tokens (extensible to 1M) |
| Multimodal | Native (text, image, and video input) |
| Speculative decoding | Built-in MTP (Multi-Token Prediction) module |
| Reasoning modes | Reasoning effort configurable as low / medium / xhigh |
| License | Qwen3.8-Max License (custom license) |
Architecturally, this is a straightforward scale-up of the hybrid design used in Qwen3.5 and Qwen3-Next. Roughly 75% of the layers are Gated DeltaNet (linear attention) and the remaining 25% are conventional GQA (64 query heads / 4 key-value heads), combined with sparse MoE and the built-in MTP module. The model introduction in the SGLang cookbook covers the details.
One detail worth noting from an inference-server parallelism standpoint: DCP (Decode Context Parallelism), which paid off substantially with Kimi-K3, is not part of the recommended configuration for Qwen3.8. Our read is that the linear-attention layers making up 75% of the model don’t accumulate much KV cache pressure even at long context, and the remaining GQA layers have only 4 KV heads, so under TP8 those heads simply get replicated twice. Compared with the MLA-centric Kimi-K3, there is much less for DCP to do.
The launch messaging is aimed squarely at coding and long-running autonomous agent work. The headline results are all long-horizon tasks: more than ten days of unattended autonomous coding, a chip design optimization run spanning roughly 500 turns, and a simulation of a full year of e-commerce operations. On the API side, Qwen offers an Anthropic-compatible interface in addition to the OpenAI-compatible one, which makes it easy to drop into existing coding agents.
A smaller Qwen3.8-27B is also scheduled for open-weight release later this week, aimed at local and edge deployment. This article covers only the flagship Qwen3.8-2.4T-A95B.
This is our second Day 1 deployment of an ultra-large open-weight model, following Kimi-K3 (2.8T) two weeks ago.
Test environment
| Item | Details |
|---|---|
| GPU | NVIDIA B300 SXM6 x8 (single node) |
| GPU memory | 275.04 GB x8 (as reported by nvidia-smi) |
| Inference engine | SGLang (lmsysorg/sglang:qwen38 image) |
| Parallelism | TP=8 |
| Speculative decoding | Built-in MTP model (NEXTN) |
We used SGLang again. Three reasons: NVFP4 quantized weights were supported from day zero (as explained below, this is what makes single-node operation possible at all); day-zero support extended as far as the DSpark draft model (which we ended up not using, since we went with the built-in MTP); and keeping the engine the same as last time lets us compare directly against Kimi-K3.
Working out the memory footprint
At 2.4T total parameters, the weights alone come to roughly 4.8TB in BF16 and roughly 2.4TB even in FP8 — neither fits in this node’s ~2.2TB of aggregate GPU memory. Running on a single node means getting down to 4-bit-class weights, around 1.2–1.5TB.
The weights Qwen distributes officially are BF16 or FP8, so neither fits as-is. We deployed using the NVFP4 quantized weights published by the SGLang team, RadixArk/Qwen3.8-2.4T-A95B-NVFP4. Amusingly, the NVFP4 version was uploaded to Hugging Face about four hours before the model itself — a good indication of how thoroughly SGLang had prepared for day-zero support. For vLLM users, there is an equivalent NVFP4 build from Inferact.
This is a meaningful difference from Kimi-K3, which shipped natively in MXFP4 with QAT already applied and therefore fit on one node out of the box. Here, post-training quantization means some accuracy loss is possible and worth watching. The model card reports Terminal-Bench 2.1 moving from 86.6% (avg@10) to 87.64% (pass@1), but with only 89 problems in that benchmark, the difference is within noise.
Deployment
Downloading the weights
The download went more smoothly than it did for Kimi-K3. The release countdown ran on ModelScope rather than Hugging Face, which may have spread the load away from Hugging Face. (We didn’t test the ModelScope route ourselves.)
We pulled the official FP8 build (~2.5TB, for comparison), the NVFP4 build (~1.48TB, used for this deployment), and the DSpark draft model (~6.6GB, ultimately unused). The NVFP4 download slowed down somewhat near the end but completed without trouble.
Launch command
We followed the recommended configuration in the SGLang cookbook.
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ${HF_HOME}:/root/.cache/huggingface \
--env SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION=1 \
--ipc=host \
lmsysorg/sglang:qwen38 \
sglang serve \
--trust-remote-code \
--model-path RadixArk/Qwen3.8-2.4T-A95B-NVFP4 \
--tp-size 8 \
--moe-runner-backend flashinfer_trtllm \
--mamba-radix-cache-strategy extra_buffer \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--enable-linear-replayssm-spec \
--mem-fraction-static 0.90 \
--chunked-prefill-size 8192 \
--max-prefill-tokens 8192 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--host 0.0.0.0 \
--port 30000
Because this is an orderly scale-up of the Qwen3 architecture, every option here is a standard one. There was no wrestling with model-specific parameters that have to be tuned per workload, as there was with Kimi-K3’s --mamba-full-memory-ratio, and TP=8 is a conventional setup. For speculative decoding, the model’s built-in MTP module is used via NEXTN.
Startup time: about 65 minutes total
Total time to a ready server was about 65 minutes, of which weight loading accounted for roughly 57.5 minutes for the main model plus 2.3 minutes for the MTP draft. That’s more than 20 minutes faster than Kimi-K3’s 88.8 minutes total (81.4 minutes of loading), but it’s still large enough that once the server is up, you don’t restart it casually.
The problem we hit last time — docker logs not showing progress during model loading — didn’t come up here, because we left docker run in the foreground. If you launch in the background, follow along with docker logs -f.
GPU memory breakdown and KV cache
Here is the GPU memory breakdown at steady state after startup (per rank, TP0 as representative).
| Item | Size (GB) | Notes |
|---|---|---|
| Total GPU memory | 275.04 | B300 SXM6 |
| In use (nvidia-smi) | 261.5 | Steady state after startup |
| Free | ~13.5 | |
| └ Model weights (main) | 167.44 | Qwen3_5MoeForCausalLM, NVFP4 (modelopt_fp4) |
| └ Model weights (MTP draft) | 7.32 | Qwen3_5ForCausalLMMTP, NVFP4 |
| └ Mamba cache (GDN state pool) | 29.40 | conv 0.83 + ssm 28.47 + conv_window 0.10 |
| └ GQA KV cache | 33.72 | K 16.86 + V 16.86; 1,537,600 tokens (bf16) |
| └ MTP KV cache | 1.46 | K 0.73 + V 0.73; 1,537,600 tokens (bf16) |
| └ Other (CUDA graph / NCCL / context, etc.) | ~22.2 | Remainder |
As with Kimi-K3, this is a hybrid model, so several distinct pools stand in for what would normally be a single KV cache.
| Pool | Capacity | Purpose |
|---|---|---|
| GQA KV | 1,537,600 tokens | For the GQA layers (~25% of the model). The primary indicator of context capacity. |
| Mamba state | 844 requests | Ceiling on concurrent requests through Gated DeltaNet |
| MTP KV | 1,537,600 tokens | KV for the speculative decoding draft |
Effective context capacity works out to about 1.54M tokens — roughly three times what we got with Kimi-K3 (about 0.53M, partly due to a memory allocation misconfiguration on our side), and achieved here with default settings. The reason is that linear attention accounts for 75% of the layers, so the KV footprint is small to begin with. KV is allocated in bf16, so switching to FP8 KV should roughly double capacity to around 3M tokens. At that point, combined with offloading, serving 1M-token contexts to multiple concurrent users starts to look practical.
Benchmark results
We measured throughput and latency under the same two workloads used in the Kimi-K3 evaluation.
Random (ISL=8K, OSL=1K, num-prompts 200)
| Concurrency | Duration (s) | Output tput (tok/s) | Total tput (tok/s) | TTFT med/p90/p99 (ms) | ITL med/p90/p99 (ms) | Failed |
|---|---|---|---|---|---|---|
| 10 | 298.0 | 687.2 | 6,184.5 | 371 / 618 / 2,745 | 20.9 / 21.6 / 271 | 0 |
| 20 | 215.6 | 950.0 | 8,549.7 | 389 / 886 / 5,443 | 27.0 / 28.3 / 279 | 0 |
| 30 | 183.2 | 1,117.7 | 10,059.3 | 401 / 3,323 / 8,159 | 31.5 / 33.8 / 284 | 0 |
| 40 | 164.1 | 1,248.2 | 11,233.5 | 414 / 6,035 / 10,852 | 35.1 / 40.9 / 288 | 0 |
| 50 | 155.8 | 1,314.8 | 11,833.1 | 2,253 / 8,740 / 25,458 | 38.3 / 93.1 / 343 | 0 |
Every row processes the same token counts: 1,638,400 in and 204,800 out. Kimi-K3 failed 119 of 200 requests at concurrency 50; here there were zero failures under any condition.
Internal coding-agent dataset (max-duration 300, multi-turn)
This time we also measured the low-concurrency end (1 / 2 / 4).
| Concurrency | Duration (s) | Output tput (tok/s) | Total tput (tok/s) | TTFT med/p90/p99 (ms) | ITL med/p90/p99 (ms) | Failed |
|---|---|---|---|---|---|---|
| 1 | 312.5 | 238.6 | 1,249.0 | 116 / 248 / 339 | 10.1 / 10.3 / 10.5 | 0 |
| 2 | 313.2 | 431.3 | 1,802.4 | 121 / 281 / 475 | 11.4 / 11.8 / 12.0 | 0 |
| 4 | 324.9 | 614.3 | 2,821.3 | 112 / 272 / 541 | 14.3 / 14.8 / 16.9 | 0 |
| 10 | 335.9 | 961.3 | 5,383.0 | 133 / 303 / 1,851 | 21.8 / 23.0 / 69 | 0 |
| 20 | 355.4 | 1,321.1 | 7,988.7 | 165 / 437 / 641 | 28.7 / 30.4 / 108 | 0 |
| 30 | 454.9 | 1,487.1 | 7,355.3 | 179 / 638 / 761 | 34.0 / 36.4 / 137 | 0 |
| 40 | 460.9 | 1,687.9 | 8,322.6 | 193 / 662 / 1,230 | 38.9 / 42.6 / 175 | 0 |
| 50 | 438.6 | 1,942.5 | 9,080.2 | 1,604 / 3,223 / 4,867 | 41.5 / 46.6 / 189 | 0 |
Throughput scaling
Left: total throughput (input + output). Right: output throughput. Measured values for Kimi-K3 on the same hardware and workload are overlaid as dotted lines for reference. There are no failures even at concurrency 50, but median TTFT jumps sharply — queueing caused by the 48-request concurrency ceiling that applies when NEXTN is enabled.
Comparison with Kimi-K3
Comparison against the 2.8T Kimi-K3 under identical hardware and load conditions:
| Item | Kimi-K3 (2.8T / 104B active) | Qwen3.8-2.4T-A95B (2.4T / 95B active) |
|---|---|---|
| Weight size (measured) | ~1.57TB (MXFP4) | ~1.48TB (NVFP4) |
| Source of quantized weights | Moonshot, official (QAT applied) | SGLang, official (RadixArk, post-training quantization) |
| Startup time (total / model load) | 88.8 min / 81.4 min | 65.3 min / 59.8 min |
| KV cache capacity (default settings) | ~0.53M tokens | ~1.54M tokens |
| Random: peak total throughput | 5,847.6 tok/s (concurrency 40) | 11,833.1 tok/s (concurrency 50) |
| Real data: peak total throughput | 3,879.6 tok/s (concurrency 30) | 9,080.2 tok/s (concurrency 50) |
| Practical concurrent request ceiling | ~30 (failures beyond that) | 48 (NEXTN concurrency limit; excess requests queue) |
Observations
It’s fast. Against Kimi-K3 under identical conditions, total throughput is roughly double. The gap is widest at low concurrency: on the real dataset at concurrency 10, it’s 5,383.0 vs. 2,477.6 tok/s, a 2.2x difference. ITL — the metric that matters most for agent workloads — is also much better, with a median of 21.8 ms at concurrency 10 against Kimi-K3’s 38.7 ms.
The scaling behavior went the other way entirely. Kimi-K3 plateaued between concurrency 30 and 40 and produced widespread failures at 50. Qwen3.8 keeps gaining total throughput all the way to concurrency 50, with no failures. What does happen at 50 is that median TTFT jumps by more than 10x. The cause is that SGLang caps concurrent requests at 48 when NEXTN (MTP) is enabled, so anything above that waits in the queue. For practical purposes, treat 48 as the concurrency ceiling.
The likely explanation for the speed gap is maturity. Qwen3.8 is an orderly scale-up of the Gated DeltaNet + GQA design that has been in production since Qwen3-Next, which means the inference libraries have had time to optimize for it. Kimi-K3 asked those same libraries to run a much more idiosyncratic architecture on day one. That difference shows up directly in the throughput numbers.
Coding performance (subjective evaluation)
We gave the model the same task as last time: implement an A* pathfinding visualizer as a single HTML file.
This round we used opencode as the harness, and ran the same prompt through DeepSeek-V4-Flash-0731 and GLM-5.2 (NVFP4) alongside Qwen3.8-2.4T-A95B so we could compare all three outputs. (For Kimi-K3’s output, see the previous article.) Because the harness differs from last time, treat the comparison with Kimi-K3 as indicative only.
Prompt used (translated in full)
Build an A* pathfinding visualizer as a single HTML file.
No external libraries, no build step — it must run by simply opening it in a browser.
[Core requirements]
- A 40x25 grid. Walls can be drawn and erased by dragging with the mouse.
- Start and goal cells can be repositioned by dragging.
- Animate the search one step at a time, color-coding to distinguish:
- the open set (candidates)
- the closed set (already explored)
- the node currently being expanded
- the final shortest path (traced back to the start and drawn once the search completes)
- Play / pause / single-step / speed slider.
[Comparison feature — required; this is the centerpiece of the demo]
- Switchable heuristics:
Manhattan distance / Euclidean distance / h=0 (i.e. Dijkstra)
- A comparison mode that puts the same board side by side in two panes
and runs both searches simultaneously with different heuristics.
- Real-time display of nodes expanded, path length, and step count.
[Implementation requirements]
- Implement the priority queue as a binary heap, not by sorting an array.
- Correctly handle reopening a node when its g score improves.
- State the tie-breaking rule for equal f values explicitly, and explain the reasoning in a comment.
- Toggle for diagonal movement (when on, cost is √2 and cutting corners through walls is forbidden).
- Additionally: right-dragging a cell sets a movement cost weight from 1 to 5,
with the weight represented by color intensity.
[UI]
Dark theme, generous whitespace, and a palette that makes search states
distinguishable at a glance. Show the controls concisely on screen.
Qwen3.8-2.4T-A95B’s output, shown on the same zigzag maze as last time in comparison mode after the search has run. It includes a random wall generator button that wasn’t requested.
DeepSeek-V4-Flash-0731’s output. All the features are there, but the play button had a bug that made it act as a single-step button.
GLM-5.2 (NVFP4)’s output. Displaying reopen-count statistics was a nice touch of its own, but the weight setting didn’t work.
Notes from using all three outputs and reading the code:
- All three hand-implemented the priority queue as a binary heap (a
MinHeapclass), as requested, and all three included at least a nominal comment. - Qwen3.8 added a random wall generator button on its own initiative, which is genuinely useful for testing. On the other hand, Kimi-K3’s approach to representing weights through color intensity was easier to read than Qwen3.8’s.
- DeepSeek-V4-Flash’s play button was broken — pressing it advanced the search by exactly one step. The cause is clear from the code: the in-progress status display calls
s.closed.size(), treating thesizeproperty of aSetas a method. That threw an exception on every step and killed the animation loop. A basic JavaScript mistake made it into the final output. - GLM-5.2 starts with obstacles already placed, which is arguably helpful, but right-drag weight editing didn’t work. Its self-directed addition of reopen-count statistics was a good idea.
Judged on getting everything working correctly on the first try, Qwen3.8 was the strongest of the three models here. But on a broader subjective read — UI polish and attention to detail included — it still falls a step short of Kimi-K3’s output from last time. Given how heavily the launch leaned on coding, we expected a decisive gap, and on this task at least, we didn’t see one.
Conclusion
A 2.4T model runs on a single node at roughly twice the throughput of Kimi-K3. That’s the main takeaway. The hybrid design built primarily on linear attention keeps the KV footprint small, allowing about 1.54M tokens of KV cache with no configuration changes, and it handles 48 concurrent requests without failures. The low ITL translates directly into a better experience for agent workloads.
From a Day 1 deployment standpoint, though, the difficulties are the mirror image of Kimi-K3’s. Because the official release is BF16 / FP8 only, single-node operation depends on third-party NVFP4 quantized weights (in this case from RadixArk, published by the SGLang team). Those were ready before the model release, which is what made Day 1 possible here — but compared with Moonshot’s approach of shipping QAT’d low-precision weights as the primary artifact, the burden of validating quantization quality sits with the user. We only checked the reference Terminal-Bench figures, so the accuracy impact of post-training quantization needs ongoing evaluation.
Operationally, the roughly 65-minute startup is more than 20 minutes faster than Kimi-K3’s, and no model-specific memory tuning was needed. Deployment was notably uneventful. The things to watch are the 48-request concurrency ceiling when NEXTN is enabled, and the usage terms of the custom Qwen3.8-Max License.
Our subjective take on coding quality was less remarkable than the speed. It produces working code without falling over, but Kimi-K3 still feels ahead on the finer details. As of now, the summary is: pick Qwen3.8 for inference speed, concurrency, and context capacity; pick Kimi-K3 for the quality of the output itself.
On the day Qwen3.8-2.4T-A95B’s weights were released, we completed a deployment to a single node of 8x NVIDIA B300 and ran benchmarks. The SGLang team’s NVFP4 weights (~1.48TB) made it fit on one node; throughput came in at roughly double Kimi-K3’s under identical conditions, with zero failures up to 48 concurrent requests. On subjective evaluation of coding output, it lands a step behind Kimi-K3 — a visible trade-off between speed and quality.
Work like this isn’t just a first look at a new model; it feeds directly into our products.
Fixstars offers Fixstars Vega, an on-premises AI appliance that pairs open-weight LLMs with our own performance engineering harness. We continuously evaluate the latest open-weight models for Vega, and this Qwen3.8-2.4T-A95B evaluation — like the Kimi-K3 one before it — is part of that work.
The goal for Vega is simple: from the day a new model is published, customers should be able to use the latest open-weight models safely in their own on-premises environment. If you’re interested in applying AI to embedded software development, take a look at the product page.
References
- Qwen3.8-2.4T-A95B (Hugging Face) / FP8 version
- RadixArk/Qwen3.8-2.4T-A95B-NVFP4 (SGLang’s official NVFP4 quantized weights)
- Qwen3.8 announcement blog (Qwen)
- SGLang cookbook: Qwen3.8
- vLLM recipes: Qwen3.8-2.4T-A95B
- Previous article: Day 0 Deployment of Kimi-K3 — Will a 2.8T Model Run on a Single Node of 8x NVIDIA B300? (First Look Benchmark)
This article was originally published in Japanese on the Fixstars Tech Blog: Qwen3.8-2.4T-A95BをDay1デプロイ!
Author
