Serving a 754B MoE Model on a Single 8-GPU Node: Optimizing GLM-5.1 Inference

TL;DR: On a single NVIDIA H200 x8 node, we cut P90 time to first token for GLM-5.1, a 754B-parameter MoE model, from 59 seconds to 3.8 seconds with 50 users. The limit was KV cache capacity, not compute: under tensor parallelism, GLM-5.1’s MLA design makes every GPU store the same KV cache. Switching to DP Attention with expert parallelism, which NVFP4 weights made possible, raised per-node KV cache capacity 3.5x. An NVIDIA B200 x8 node met every target at 90 users, while H200 missed the 50 ms per-token target by about 4 ms.

Running a large open-weight LLM on your own hardware keeps code and data inside the company. The hard part is making it fast enough for many people to use at once.

We set out to serve GLM-5.1, a 754B-parameter mixture-of-experts (MoE) model, on a single 8-GPU node for an on-premises deployment. The model fits on one NVIDIA H200 x8 node, but in its default configuration the 90th-percentile time to first token was 59 seconds with 50 users. After the changes described below, it dropped to 3.8 seconds, and per-node KV cache capacity grew 3.5x. On an NVIDIA B200 x8 node, every target was met even at 90 concurrent users.

We presented this work at the 2026 IEEE International Conference on Device Technologies for Diversified Applications (IEEE DTDA 2026) in Sapporo, Japan, on October 14, 2026, in the talk “Memory Capacity, Not Compute, Bounds Large-MoE Serving: A 15x Latency Reduction on GLM-5.1”. This post explains why the default setup falls short, what we changed, and what it means if you are choosing hardware for a large MoE model.

The constraint and the targets

The deployment allowed one 8-GPU node, with no scaling out across multiple nodes. We tested on two systems:

• NVIDIA H200 x8: 1,128 GB of GPU memory
• NVIDIA B200 x8: 1,440 GB of GPU memory

We set the following service targets:

• Concurrency: 50 users
• P90 time to first token (TTFT): 30 seconds or less
• P90 time per output token (TPOT): 50 ms or less, which works out to about 20 tokens per second per user

Why the default setup falls short

GLM-5.1 combines two design choices that matter for serving. Almost all of its parameters (about 725B of 754B) sit in a very sparse MoE layer: 1 shared expert and 256 routed experts, with only 8 routed experts active per token. Its attention uses Multi-head Latent Attention (MLA), which compresses the keys and values for every head into a single latent vector per token, 512 + 64 values per token per layer.

MLA keeps the KV cache small per token, but it causes a problem under tensor parallelism (TP), the usual way to split a model across GPUs. With standard multi-head or grouped-query attention, each GPU holds the keys and values for its own heads, so each GPU stores one-eighth of the KV cache. With MLA, every head is built from the same shared latent, so each of the eight GPUs has to store a full copy.

The result: on the H200 node, the weights take only about 470 GB of the 1,128 GB available, but the remaining memory has to hold eight identical copies of the KV cache. That leaves room for only 679,808 tokens across the whole node, the equivalent of one GPU. Once that fills up, older context is evicted, and the next request that needs it has to recompute it from scratch (re-prefill). This is what drives the 59-second TTFT.

Change 1: Data-parallel attention with expert parallelism

Instead of splitting every layer with TP=8, we ran attention in data-parallel mode (DP Attention) and spread the MoE layer with expert parallelism (EP). Each GPU now handles its own requests and keeps its own independent KV cache, and each GPU holds 32 of the 256 routed experts. Tokens are exchanged between GPUs with NCCL all-to-all communication.

The trade-off is that attention weights and the shared expert are now copied to every GPU. With NVFP4 weights, that raises per-GPU weight memory from 55.1 GB to 83.8 GB, or from 441 GB to 671 GB across the node.

Even so, the node now holds eight independent caches instead of eight copies of one. On H200, KV cache capacity grew from 679,808 tokens to about 2.4 million tokens per node, 3.5x the original. On B200, the extra memory goes to KV cache, reaching about 6 million tokens per node, roughly 2.5x the optimized H200.

This only worked because of NVFP4 quantization. With the FP8 checkpoint, the weights alone would need about 850 GB under DP Attention, which leaves little room for KV cache on a 1,128 GB node. According to the model card for the NVFP4 version of GLM-5.1, accuracy stays close to FP8: GPQA Diamond goes from 85.61 to 85.02, AIME 2026 stays at 96.67, and IFBench goes from 76.56 to 76.33.

Change 2: Cache-aware routing and CPU offload

Data parallelism introduces a new risk. If a request lands on a GPU that does not hold its earlier context, that GPU rebuilds a KV cache that already exists elsewhere. We addressed this in two ways:

• KV-cache-aware request routing: each request goes to the GPU that already holds its prefix.
• CPU cache offload: evicted KV cache is kept in CPU memory and read back, instead of being recomputed.

This matters most for agent workloads. Coding agents send long contexts over many turns, and every prefill that has to be recomputed adds directly to TTFT.

Testing against real usage

Synthetic benchmarks often look good and then disappoint in production. To avoid that, we built the benchmark from 260 real sessions in which our own engineers used coding agents in their daily work, on Fixstars Vega, our in-house AI platform.

We analyzed the input and output token lengths per turn and the number of turns per conversation, fit lognormal distributions to the input and output lengths, and replayed multi-turn conversations from a dataset generated from those statistics. Input lengths per turn ranged up to about 20,000 tokens, and some conversations ran close to 40 turns.

Results

All results are P90 values from this benchmark on a single node, using GLM-5.1-NVFP4.

On H200, TTFT with 50 users fell from 59 seconds to 3.8 seconds, about one-fifteenth, and stayed well within the 30-second target at 90 users as well. TPOT improved from 63 ms to 54 ms but remained slightly above the 50 ms target. On B200, both TTFT and TPOT met the targets at 50 and 90 users, well beyond the original goal of 50.

What this means when choosing hardware

Two lessons from this project apply to anyone picking an inference platform for a large MoE model.

Ask how much KV cache is left, not whether the model fits. Fitting the weights is not the same as fitting your users. On the same hardware, the choice between TP and DP with EP changed the KV cache ceiling several times over.

Choose quantization and parallelism together. NVFP4 was not just a way to save memory here. It was what made DP Attention possible in the first place.

How much this helps depends on how a model stores keys and values, not on its name. Models built on MLA, including the DeepSeek V3 and V4 families, Kimi K2.6 and K3, and Mistral Large 3, should benefit the most. DeepSeek V4 and Kimi K3 compress their KV cache further, so the gain there will be smaller than with GLM-5.1. Models that use grouped-query attention, such as Llama and Qwen3, already split their KV cache by head under TP, so they gain less. For how two of these MLA models run on a single node, see our Day 1 posts on Kimi-K3 and DeepSeek-V4-Pro-0813.

What comes next

Next, we plan to disaggregate prefill and decode, so that each side handles a simpler, more predictable workload. We will also evaluate options that trade a little accuracy for more capacity, such as an FP8 KV cache and an indexer cache, and validate the method on more GPUs and on workloads beyond coding agents.

Get the full details

None of this came from a single trick. It came from a repeated cycle: measure on the real workload, find the bottleneck, choose a fix that fits the model’s architecture, and measure again.

Our white paper, Optimizing Inference Performance for the Open-Weight LLM GLM-5.1, covers the exact configurations, the full measurement tables, and the appendices behind these results. If you are planning to run large open-weight models on your own hardware, talk to our performance engineers.

FAQ