Back previous/next report: k j
NVIDIA GB300 NVL72 (4-GPU SUT)
Submitted by NVIDIAon 2026-10-15 Published on 2026-11-04
Result statement
NVIDIA ran deepseek-v4_1-flash on NVIDIA GB300 NVL72 (4-GPU SUT) and achieved a verified 55,137.8 tokens/second System TPS at 19,889.5ms TTFT P90 in the Standardized division of the Available category on November 4, 2026 .
[1] MLPerf Endpoints v1.0 deepseek-v4_1-flash, published November 4, 2026; retrieved from endpoints.mlcommons.org on November 7, 2026. Result —, run 9: concurrency = 2,048 queries, TTFT P90 = 19,889.5ms, System TPS = 55,137.8 tokens/sec, interactivity = 26.9 tokens/sec per user and total provisioned system power = 7.5 kW. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.
NVIDIA
c0b99dd8-3157-4985-a669-acb2a6659c2a
Submission Status PUBLISHED
2026-10-15
2026-11-04
Division Standardized
Deployment Client on Premises
Model Name deepseek-v4_1-flash
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Dataset Name mlperf_agentic_inference_dataset
Dataset Type performance
Average Input Tokens —
Dataset Link —
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P90 vs. Concurrency
Tokens/s per User vs. Concurrency
NVIDIA GB300 NVL72 (4-GPU SUT)
Available
System Category — · —
4 accelerators
1
1
Max Supported Concurrency 2,048
Node Type ID 0
Node Count 1
Framework vLLM 0.20.2rc1.dev6106+g7cf5542c2.d20261003 behind Dynamo
vLLM 0.20.2rc1.dev6106+g7cf5542c2.d20261003 behind Dynamo
vLLM behind a Dynamo frontend
580.173.10
Link to Container registry.gitlab.com/nvidia/mlperf-inference-partner/nv-mlpinf-partner/centml-rubin-vllm:dsv41-rubin-7cf5542c25-r1
Operating System unspecified
Filesystem unspecified
Other Software unspecified
Software Notes —
Processor
NVIDIA Grace (ARM Neoverse-V2)
2
144
VCPUs Per Processor —
Accelerator
—
0
—
—
—
—
Host / Storage
unspecified
Memory Configuration unspecified
Storage Capacity unspecified
Storage Type unspecified
Networking
unspecified
NVLink + InfiniBand
Network Topology —
Other
Other Hardware —
Cooling Liquid-cooled
Hardware Notes —
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 |
|---|---|---|---|---|---|---|---|---|---|
| Run Date | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 | 10/15/2026 |
| Concurrency | 16 | 24 | 32 | 64 | 128 | 256 | 512 | 1,024 | 2,048 |
| System Tokens/Second | 3,054.0 | 4,150.1 | 5,184.3 | 8,895.4 | 14,287.1 | 22,093.8 | 35,185.2 | 49,701.8 | 55,137.8 |
| Tokens/Second per User | 190.9 | 172.9 | 162.0 | 139.0 | 111.6 | 86.3 | 68.7 | 48.5 | 26.9 |
| TTFT P90 (ms) | 112.8 | 115.4 | 120.3 | 133.2 | 157.7 | 203.0 | 310.8 | 4073.5 | 19889.5 |
| Utilization | 5.5% | 7.5% | 9.4% | 16.1% | 25.9% | 40.1% | 63.8% | 90.1% | 100.0% |
| Configuration Summary | deepseek-v4.1-flash-nvflow · online · c16 | deepseek-v4.1-flash-nvflow · online · c24 | deepseek-v4.1-flash-nvflow · online · c32 | deepseek-v4.1-flash-nvflow · online · c64 | deepseek-v4.1-flash-nvflow · online · c128 | deepseek-v4.1-flash-nvflow · online · c256 | deepseek-v4.1-flash-nvflow · online · c512 | deepseek-v4.1-flash-nvflow · online · c1024 | deepseek-v4.1-flash-nvflow · online · c2048 |
| Config Type | online | online | online | online | online | online | online | online | online |
| Config Region | — | — | — | — | — | — | — | — | — |
| Target QPS | — | — | — | — | — | — | — | — | — |
| Result State | complete | complete | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — | — | — |
| Benchmark Version | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown | unknown |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/e315148a-75aa-4572-bcc1-e8e9a9d7b8e3/archive.tar.gz | gs://mlperf-endpoints-blob/runs/219d21c1-dd83-45f1-88e3-bc650572b3c0/archive.tar.gz | gs://mlperf-endpoints-blob/runs/67d883f3-37c5-419c-a2c4-177b83710fce/archive.tar.gz | gs://mlperf-endpoints-blob/runs/34c72228-626d-4ca3-b082-ff1f73fc7180/archive.tar.gz | gs://mlperf-endpoints-blob/runs/d2ba72d0-19a8-4475-9d33-53dd9e93384c/archive.tar.gz | gs://mlperf-endpoints-blob/runs/2d23fe07-e9b1-44b5-84c3-a54b0db327d9/archive.tar.gz | gs://mlperf-endpoints-blob/runs/2089aadd-d1d0-4083-8d74-b059c10f4d31/archive.tar.gz | gs://mlperf-endpoints-blob/runs/79639808-fde6-4d61-b9f9-82dd707bfa82/archive.tar.gz | gs://mlperf-endpoints-blob/runs/e4ab943c-86dd-4e74-8568-07ce4e755f87/archive.tar.gz |
| QPS | 3.3285 | 4.5723 | 5.6055 | 9.6819 | 15.3639 | 24.3776 | 38.1899 | 52.3344 | 50.0526 |
| Total Output Tokens | 18,018,310 | 17,437,885 | 17,520,332 | 35,985,349 | 34,554,443 | 67,976,180 | 136,427,661 | 120,808,935 | 149,384,321 |
| Average Output Tokens | 917.52 | 907.66 | 924.85 | 918.77 | 929.91 | 906.31 | 921.32 | 949.7 | 1,101.6 |
| Run Duration (s) | 5,899.88 | 4,201.81 | 3,379.51 | 4,045.38 | 2,418.58 | 3,076.71 | 3,877.41 | 2,430.68 | 2,709.29 |
| Total Queries | 19,638 | 19,212 | 18,944 | 39,167 | 37,159 | 75,003 | 148,078 | 127,208 | 135,607 |
| Total Queries Issued | 19,638 | 19,212 | 18,944 | 39,167 | 37,159 | 75,003 | 148,078 | 127,208 | 135,607 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) (ms) | |||||||||
| Minimum | 34.9 | 37.4 | 39.4 | 43.5 | 47.7 | 56.6 | 75.3 | 118.9 | 559.0 |
| Average | 78.6 | 80.8 | 86.2 | 96.5 | 121.0 | 158.7 | 249.5 | 2396.7 | 17399.6 |
| P50 | 70.5 | 71.9 | 74.7 | 84.2 | 98.2 | 126.1 | 173.0 | 2075.1 | 16902.2 |
| P90 | 112.8 | 115.4 | 120.3 | 133.2 | 157.7 | 203.0 | 310.8 | 4073.5 | 19889.5 |
| P95 | 130.0 | 132.5 | 138.9 | 155.4 | 187.7 | 242.6 | 417.5 | 4871.8 | 21093.4 |
| P99 | 193.4 | 205.5 | 255.7 | 248.1 | 361.0 | 392.5 | 924.1 | 12301.9 | 38290.7 |
| P999 | 895.3 | 980.2 | 1534.3 | 1587.9 | 4193.6 | 8184.5 | 14197.9 | 29340.6 | 61063.0 |
| Maximum | 2853.9 | 1603.9 | 1818.1 | 3359.3 | 7768.8 | 9458.7 | 17758.5 | 32693.3 | 65495.8 |
| Time Per Output Token (TPOT) (ms) | |||||||||
| Minimum | 1.15 | 1.19 | 1.37 | 1.66 | 1.98 | 0.07 | 0.03 | 0.02 | -0.00 |
| Average | 2.77 | 3.17 | 3.49 | 4.31 | 5.62 | 7.53 | 9.74 | 12.1 | 13.5 |
| P50 | 2.58 | 2.94 | 3.23 | 4.00 | 5.15 | 6.95 | 8.98 | 11.3 | 12.9 |
| P90 | 4.00 | 4.62 | 5.08 | 6.28 | 8.28 | 11.3 | 14.6 | 18.0 | 20.0 |
| P95 | 4.40 | 5.04 | 5.55 | 6.82 | 9.02 | 12.4 | 15.9 | 19.5 | 21.7 |
| P99 | 5.25 | 6.07 | 6.77 | 8.26 | 11.2 | 15.0 | 18.5 | 22.7 | 25.3 |
| P999 | 7.85 | 9.15 | 14.2 | 16.5 | 24.8 | 22.6 | 22.7 | 29.2 | 35.1 |
| Maximum | 18.9 | 23.7 | 20.9 | 61.8 | 86.7 | 457.0 | 85.0 | 116.8 | 131.1 |
| Query Latency (ms) | |||||||||
| Minimum | 97.0 | 85.1 | 99.2 | 124.3 | 164.0 | 146.5 | 188.0 | 422.5 | 10677.6 |
| Average | 3178.7 | 3599.9 | 4016.4 | 4962.8 | 6549.1 | 8670.1 | 11488.0 | 16763.1 | 35934.0 |
| P50 | 555.6 | 628.5 | 689.5 | 832.3 | 1072.9 | 1423.7 | 1893.7 | 4853.2 | 20329.1 |
| P90 | 7628.0 | 8547.8 | 9613.2 | 11717.7 | 15336.9 | 20164.1 | 27019.8 | 37738.9 | 74401.0 |
| P95 | 21170.5 | 22573.9 | 28191.6 | 34380.1 | 48009.4 | 59707.8 | 81462.9 | 121286.2 | 161376.0 |
| P99 | 33980.9 | 39120.9 | 42649.4 | 53425.0 | 70290.3 | 95724.2 | 124009.2 | 154967.2 | 183860.3 |
| P999 | 37738.6 | 41925.6 | 45841.3 | 56944.0 | 75148.1 | 103380.8 | 132204.1 | 165863.3 | 200003.3 |
| Maximum | 42070.3 | 48466.4 | 49812.6 | 62988.2 | 81614.7 | 111996.5 | 143417.1 | 178280.1 | 220182.9 |