Back previous/next report: k j
NVIDIA GB300 NVL72
Submitted by NVIDIAon 2026-06-25
Result statement
NVIDIA ran GPT-OSS 120B on NVIDIA GB300 NVL72 and achieved a verified 954,230.2 tokens/second System TPS at 1,170.4ms TTFT P90 in the Standardized division of the Available category .
[1] MLPerf Endpoints v1.0 GPT-OSS 120B; retrieved from endpoints.mlcommons.org on October 22, 2026. Result —, run 14: concurrency = 22,528 queries, TTFT P90 = 1,170.4ms, System TPS = 954,230.2 tokens/sec and interactivity = 42.4 tokens/sec per user. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.
NVIDIA
1907bca4-63a0-4b60-ba97-ef47b657d195
Submission Status PUBLISHED
2026-06-25
—
Division Standardized
Deployment Client on Premises
Model Name GPT-OSS 120B
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Dataset Name perf
Dataset Type performance
Average Input Tokens —
Dataset Link —
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P90 vs. Concurrency
Tokens/s per User vs. Concurrency
NVIDIA GB300 NVL72
Available
System Category Datacenter · GB300 NVL72 rack, disaggregated TRT-LLM, MXFP4 weights
—
1
18
Max Supported Concurrency 22,528
Node Type ID 0
Node Count 18
Framework TensorRT-LLM
TensorRT-LLM
TensorRT-LLM
—
Link to Container —
Operating System unspecified
Filesystem —
Other Software —
Software Notes —
Processor
NVIDIA Grace (ARM Neoverse V2)
1
72
VCPUs Per Processor —
Accelerator
NVIDIA GB300
4
—
288 GB HBM3e
NVLink 5 + NVSwitch (NVL72)
NVLink-C2C (Grace-Blackwell)
Host / Storage
unspecified
Memory Configuration —
Storage Capacity unspecified
Storage Type unspecified
Networking
—
NVLink + InfiniBand
Network Topology NVL72 single-rack NVLink domain
Other
Other Hardware —
Cooling Liquid-cooled
Hardware Notes —
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 | Run 11 | Run 12 | Run 13 | Run 14 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Run Date | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 | 06/25/2026 |
| Concurrency | 8 | 17 | 35 | 70 | 140 | 272 | 544 | 1,024 | 2,048 | 3,840 | 7,168 | 12,288 | 12,800 | 22,528 |
| System Tokens/Second | 3,226.6 | 6,684.7 | 12,934.2 | 23,078.1 | 41,092.7 | 71,693.1 | 125,323.6 | 192,455.6 | 303,515.8 | 441,207.9 | 628,875.2 | 778,099.7 | 797,819.5 | 954,230.2 |
| Tokens/Second per User | 403.3 | 393.2 | 369.5 | 329.7 | 293.5 | 263.6 | 230.4 | 187.9 | 148.2 | 114.9 | 87.7 | 63.3 | 62.3 | 42.4 |
| TTFT P90 (ms) | 135.8 | 98.3 | 101.8 | 110.2 | 159.6 | 614.2 | 462.5 | 710.5 | 658.6 | 693.3 | 4545.9 | 646.7 | 806.2 | 1170.4 |
| Utilization | 0.3% | 0.7% | 1.4% | 2.4% | 4.3% | 7.5% | 13.1% | 20.2% | 31.8% | 46.2% | 65.9% | 81.5% | 83.6% | 100.0% |
| Configuration Summary | gpt-oss-120b-sil_tep8_sys8_ng8_nc8_g1_c1_fe1 · online · c8 | gpt-oss-120b-sil_tep4_sys17_ng17_nc4_g1_c1_fe1 · online · c17 | gpt-oss-120b-sil_tep2_sys35_ng35_nc2_g1_c1_fe1 · online · c35 | gpt-oss-120b-sil_tep2_sys70_ng35_nc2_g2_c1_fe1 · online · c70 | gpt-oss-120b-sil_tep2_sys140_ng35_nc2_g4_c2_fe1 · online · c140 | gpt-oss-120b-sil_tep4_sys272_ng17_nc4_g16_c2_fe1 · online · c272 | gpt-oss-120b-sil_tep4_sys544_ng17_nc4_g32_c4_fe1 · online · c544 | gpt-oss-120b-sil_tep4_sys1024_ng16_nc8_g64_c2_fe16 · online · c1024 | gpt-oss-120b-sil_dep8_sys2048_ng8_nc8_g256_c16_fe8 · online · c2048 | gpt-oss-120b-sil_dep4_sys3840_ng15_nc12_g256_c8_fe15 · online · c3840 | gpt-oss-120b-sil_dep8_sys7168_ng7_nc16_g1024_c32_fe7 · online · c7168 | gpt-oss-120b-sil_dep2_sys12288_ng25_nc22_g512_c8_fe25 · online · c12288 | gpt-oss-120b-sil_dep2_sys12800_ng25_nc22_g512_c8_fe25 · online · c12800 | gpt-oss-120b-sil_tep1_sys22528_ng48_nc24_g512_c32_fe48 · online · c22528 |
| Config Type | online | online | online | online | online | online | online | online | online | online | online | online | online | online |
| Config Region | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Target QPS | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Result State | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| Benchmark Version | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc | 629c1bc |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/cad766b3-9ddc-43f4-b3b7-d3e7fb77682b/archive.tar.gz | gs://mlperf-endpoints-blob/runs/f33b91e1-bc12-4ad7-924d-040f209814a1/archive.tar.gz | gs://mlperf-endpoints-blob/runs/4abe5c9c-6f73-4932-8bc9-18bb33783b10/archive.tar.gz | gs://mlperf-endpoints-blob/runs/88c4775c-259c-4702-a325-2099bbd217b6/archive.tar.gz | gs://mlperf-endpoints-blob/runs/d18a2c56-620e-4e13-8789-2ac6e18b1cba/archive.tar.gz | gs://mlperf-endpoints-blob/runs/8acf867c-9a7c-4430-8839-1ddf462d057c/archive.tar.gz | gs://mlperf-endpoints-blob/runs/8666872a-2d6a-481c-b3eb-d22fa6b39a0b/archive.tar.gz | gs://mlperf-endpoints-blob/runs/736ea21d-18f6-437c-9aa1-5777550e2ee4/archive.tar.gz | gs://mlperf-endpoints-blob/runs/655970e0-1055-419f-be6b-ccbb11ff87b7/archive.tar.gz | gs://mlperf-endpoints-blob/runs/e77820f2-0af4-41da-977e-d6685bbe23d0/archive.tar.gz | gs://mlperf-endpoints-blob/runs/4d0be1c5-ead7-47d4-9ce0-8e1fa40bdfcd/archive.tar.gz | gs://mlperf-endpoints-blob/runs/20e46720-5c96-4cf0-bd03-a39ca7b985bb/archive.tar.gz | gs://mlperf-endpoints-blob/runs/6d2df9ef-ccd1-4331-ab57-37c7058ab993/archive.tar.gz | gs://mlperf-endpoints-blob/runs/1c6cf0dd-dcda-4835-9ab3-54475b13d533/archive.tar.gz |
| QPS | 2.6376 | 5.4442 | 10.5227 | 18.7475 | 33.3943 | 58.2258 | 101.8297 | 156.4203 | 246.5849 | 358.7057 | 511.1075 | 632.2840 | 648.4210 | 775.5894 |
| Total Output Tokens | 2,723,113 | 5,513,088 | 10,798,227 | 19,469,442 | 36,811,323 | 64,381,935 | 112,256,121 | 173,690,797 | 266,528,001 | 394,994,669 | 548,465,566 | 659,982,592 | 666,236,494 | 796,780,859 |
| Average Output Tokens | 1,223.32 | 1,227.86 | 1,229.17 | 1,231 | 1,230.53 | 1,231.29 | 1,230.72 | 1,230.37 | 1,230.88 | 1,230 | 1,230.42 | 1,230.62 | 1,230.4 | 1,230.33 |
| Run Duration (s) | 843.96 | 824.73 | 834.86 | 843.63 | 895.81 | 898.02 | 895.73 | 902.50 | 878.14 | 895.26 | 872.14 | 848.20 | 835.07 | 835.00 |
| Total Queries | 2,226 | 4,490 | 8,785 | 15,816 | 29,915 | 52,288 | 91,212 | 141,169 | 216,535 | 321,134 | 445,756 | 536,302 | 541,478 | 647,616 |
| Total Queries Issued | 2,226 | 4,490 | 8,785 | 15,816 | 29,915 | 52,288 | 91,212 | 141,169 | 216,535 | 321,134 | 445,756 | 536,302 | 541,478 | 647,616 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) (ms) | ||||||||||||||
| Minimum | 73.9 | 27.4 | 27.7 | 28.5 | 28.4 | 30.1 | 38.4 | 36.9 | 36.9 | 39.2 | 52.9 | 52.7 | 54.9 | 61.1 |
| Average | 137.7 | 87.4 | 89.9 | 89.9 | 111.6 | 244.1 | 288.4 | 398.4 | 457.6 | 515.5 | 2053.4 | 639.6 | 723.4 | 945.8 |
| P50 | 105.4 | 62.3 | 62.8 | 66.6 | 86.6 | 101.9 | 237.5 | 333.8 | 390.5 | 458.9 | 1303.4 | 471.1 | 493.9 | 371.0 |
| P90 | 135.8 | 98.3 | 101.8 | 110.2 | 159.6 | 614.2 | 462.5 | 710.5 | 658.6 | 693.3 | 4545.9 | 646.7 | 806.2 | 1170.4 |
| P95 | 156.7 | 112.8 | 119.5 | 128.1 | 190.1 | 885.2 | 568.4 | 877.3 | 782.3 | 799.2 | 6115.8 | 752.4 | 1025.8 | 2472.0 |
| P99 | 1929.7 | 1003.9 | 1039.9 | 197.4 | 293.6 | 1568.4 | 893.1 | 1282.9 | 1198.5 | 2897.5 | 9945.8 | 9441.6 | 10023.1 | 15715.4 |
| P999 | 2176.2 | 2346.4 | 2472.5 | 2752.0 | 3122.1 | 3782.1 | 4688.5 | 4140.6 | 7052.2 | 9497.2 | 15657.1 | 15539.7 | 16258.0 | 27075.1 |
| Maximum | 2262.0 | 2435.9 | 2546.2 | 2875.5 | 3798.5 | 4022.2 | 6165.1 | 4936.4 | 7892.7 | 10793.5 | 16244.2 | 17152.8 | 17855.3 | 28043.3 |
| Time Per Output Token (TPOT) (ms) | ||||||||||||||
| Minimum | 1.43 | 1.37 | 1.30 | 1.65 | 1.69 | 2.16 | 2.14 | 3.02 | 4.30 | 4.92 | 4.56 | 7.39 | 7.31 | 8.56 |
| Average | 2.39 | 2.46 | 2.62 | 2.95 | 3.30 | 3.58 | 4.09 | 4.98 | 6.32 | 8.22 | 9.58 | 15.1 | 15.2 | 22.4 |
| P50 | 2.39 | 2.47 | 2.62 | 2.95 | 3.29 | 3.56 | 4.08 | 4.98 | 6.31 | 8.26 | 9.58 | 15.1 | 15.3 | 22.5 |
| P90 | 2.43 | 2.53 | 2.73 | 3.10 | 3.44 | 3.73 | 4.17 | 5.04 | 6.45 | 8.37 | 9.79 | 15.2 | 15.4 | 23.0 |
| P95 | 2.44 | 2.56 | 2.76 | 3.14 | 3.48 | 3.74 | 4.20 | 5.05 | 6.49 | 8.39 | 9.84 | 15.3 | 15.4 | 23.1 |
| P99 | 2.46 | 2.61 | 2.84 | 3.24 | 3.55 | 3.77 | 4.24 | 5.08 | 6.56 | 8.45 | 9.95 | 15.4 | 15.5 | 23.3 |
| P999 | 2.50 | 2.66 | 2.94 | 3.36 | 3.66 | 3.78 | 4.29 | 5.11 | 6.80 | 8.48 | 9.99 | 15.4 | 15.6 | 23.5 |
| Maximum | 2.79 | 2.68 | 3.16 | 4.26 | 5.27 | 4.37 | 4.37 | 5.95 | 7.36 | 13.6 | 18.5 | 23.9 | 31.4 | 36.5 |
| Query Latency (ms) | ||||||||||||||
| Minimum | 1276.0 | 1289.0 | 1254.7 | 1437.9 | 1526.9 | 1753.5 | 2077.3 | 2557.7 | 674.1 | 3869.4 | 4718.6 | 6827.5 | 1216.9 | 7716.8 |
| Average | 3025.8 | 3114.4 | 3319.3 | 3723.2 | 4175.2 | 4654.5 | 5318.5 | 6514.9 | 8226.3 | 10616.0 | 13827.2 | 19147.9 | 19424.3 | 28436.1 |
| P50 | 2996.9 | 3058.5 | 3261.6 | 3663.0 | 4111.3 | 4572.7 | 5245.4 | 6437.4 | 8118.6 | 10463.1 | 13467.7 | 18845.5 | 19111.0 | 27928.3 |
| P90 | 3806.4 | 3907.6 | 4200.2 | 4727.2 | 5230.0 | 5863.8 | 6621.0 | 8109.4 | 10226.6 | 13213.7 | 17780.6 | 23958.1 | 24305.8 | 35742.1 |
| P95 | 4063.2 | 4212.5 | 4531.4 | 5067.6 | 5628.3 | 6317.0 | 7088.5 | 8671.5 | 10945.4 | 14165.2 | 19425.9 | 25804.1 | 26156.9 | 38610.5 |
| P99 | 4691.1 | 4970.4 | 5194.4 | 5847.5 | 6483.1 | 7365.5 | 8211.0 | 9919.6 | 12728.8 | 16614.7 | 23100.0 | 30546.3 | 31064.8 | 46507.8 |
| P999 | 5543.1 | 5828.4 | 6168.6 | 7236.8 | 7879.3 | 8578.6 | 10185.7 | 11864.8 | 15836.8 | 20800.8 | 28044.7 | 37447.3 | 38081.6 | 57631.6 |
| Maximum | 6144.3 | 6781.1 | 7099.6 | 8216.9 | 9591.8 | 10540.3 | 12881.9 | 16847.3 | 25519.3 | 28471.9 | 37424.7 | 66361.9 | 60524.8 | 93108.6 |