Back previous/next report: k j
1-node-2S-GNR_128C
Submitted by Intelon 2026-06-26
Result statement
Intel ran Llama 3.1 8B on 1-node-2S-GNR_128C and achieved a verified 1,767.3 tokens/second System TPS at 106,156.8ms TTFT P90 in the Standardized division of the Available category .
[1] MLPerf Endpoints v1.0 Llama 3.1 8B; retrieved from endpoints.mlcommons.org on October 22, 2026. Result —, run 10: concurrency = 3,072 queries, TTFT P90 = 106,156.8ms, System TPS = 1,767.3 tokens/sec and interactivity = 0.6 tokens/sec per user. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.
Intel
f82a98ec-b5f8-42ea-b1c4-ded6f25a822b
Submission Status PUBLISHED
2026-06-26
—
Division Standardized
Deployment Client on Premises
Model Name Llama 3.1 8B
Division Standardized
Model Precision FP16
Model Link Hugging Face
Calibration Link GitHub
Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P90 vs. Concurrency
Tokens/s per User vs. Concurrency
1-node-2S-GNR_128C
Available
System Category Datacenter · —
1x2S-GNR-AP
1
1
Max Supported Concurrency 3,072
Node Type ID 0
Node Count 1
Framework vLLM 0.23.0
vLLM 0.23.0
Not detected: CUDA/ROCm/XPU runtime not found
Not available
Link to Container —
Operating System centos 9
Filesystem ext4 iso9660 ramfs vfat xfs
Other Software —
Software Notes —
Processor
Intel(R) Xeon(R) 6980P
2
256
VCPUs Per Processor 256
Accelerator
Not available
0
Not available
Not available
Not available
Not available
Host / Storage
2.3T
Memory Configuration 2304GB (24x96GB DDR5 8800 MT/s)
Storage Capacity 4.9 TB NVMe SSD, 243 GB SSD
Storage Type NVMe SSD
Networking
1x Ethernet
Ethernet
Network Topology —
Other
Other Hardware —
Cooling —
Hardware Notes —
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Run Date | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 |
| Concurrency | 3 | 16 | 40 | 128 | 256 | 512 | 1,024 | 1,536 | 2,048 | 3,072 |
| System Tokens/Second | 96.5 | 366.2 | 648.8 | 1,045.1 | 1,306.1 | 1,479.3 | 1,607.4 | 1,636.3 | 1,742.2 | 1,767.3 |
| Tokens/Second per User | 32.2 | 22.9 | 16.2 | 8.2 | 5.1 | 2.9 | 1.6 | 1.1 | 0.9 | 0.6 |
| TTFT P90 (ms) | 453.6 | 543.4 | 589.2 | 1119.9 | 1493.7 | 2064.1 | 4583.2 | 14332.5 | 45042.0 | 106156.8 |
| Utilization | 5.5% | 20.7% | 36.7% | 59.1% | 73.9% | 83.7% | 91.0% | 92.6% | 98.6% | 100.0% |
| Configuration Summary | llama-3.1-8b-concurrency · submission · c3 | llama-3.1-8b-concurrency · submission · c16 | llama-3.1-8b-concurrency · submission · c40 | llama-3.1-8b-concurrency · submission · c128 | llama-3.1-8b-concurrency · submission · c256 | llama-3.1-8b-concurrency · submission · c512 | llama-3.1-8b-concurrency · submission · c1024 | llama-3.1-8b-concurrency · submission · c1536 | llama-3.1-8b-concurrency · submission · c2048 | llama-3.1-8b-concurrency · submission · c3072 |
| Config Type | submission | submission | submission | submission | submission | submission | submission | submission | submission | submission |
| Config Region | — | — | — | — | — | — | — | — | — | — |
| Target QPS | — | — | — | — | — | — | — | — | — | — |
| Result State | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — | — | — | — |
| Benchmark Version | bbc8697 | bbc8697 | bbc8697 | bbc8697 | c81001b | c81001b | c81001b | bbc8697 | c81001b | c81001b |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/f2ed1200-7d9f-4a09-b222-0d0b6533c440/archive.tar.gz | gs://mlperf-endpoints-blob/runs/fb68bc14-e88f-4803-8d67-9ea61be1544a/archive.tar.gz | gs://mlperf-endpoints-blob/runs/802201ce-3015-4c24-8401-37b1b8025048/archive.tar.gz | gs://mlperf-endpoints-blob/runs/08cd248c-087f-486e-b173-d58df0a1ff3c/archive.tar.gz | gs://mlperf-endpoints-blob/runs/72e238f4-b53c-428d-8b53-1b6a9da0b690/archive.tar.gz | gs://mlperf-endpoints-blob/runs/4b98ac30-88a4-41e3-ba5e-c8a1abc6fd27/archive.tar.gz | gs://mlperf-endpoints-blob/runs/f8b8081e-49f6-4a84-a1eb-6cc0b084dd63/archive.tar.gz | gs://mlperf-endpoints-blob/runs/d16a14db-3524-4281-aca7-02e158498689/archive.tar.gz | gs://mlperf-endpoints-blob/runs/91dcd2dd-996a-4629-8280-245790ecfb17/archive.tar.gz | gs://mlperf-endpoints-blob/runs/dbef15d2-82c9-4546-b316-09e2b8d7acd3/archive.tar.gz |
| QPS | 0.7541 | 2.8608 | 5.0690 | 8.1645 | 10.2036 | 11.5571 | 12.5578 | 12.7835 | 13.6109 | 13.8067 |
| Total Output Tokens | 1,711,103 | 1,711,101 | 1,711,101 | 1,711,104 | 1,711,102 | 1,711,102 | 1,711,099 | 1,711,102 | 1,711,102 | 1,711,103 |
| Average Output Tokens | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
| Run Duration (s) | 17,726.31 | 4,672.83 | 2,637.19 | 1,637.33 | 1,310.12 | 1,156.69 | 1,064.52 | 1,045.72 | 982.15 | 968.22 |
| Total Queries | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 |
| Total Queries Issued | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) (ms) | ||||||||||
| Minimum | 85.9 | 89.5 | 105.1 | 135.5 | 169.3 | 227.8 | 240.0 | 225.1 | 233.2 | 267.1 |
| Average | 285.8 | 337.1 | 368.4 | 702.7 | 1051.9 | 1817.2 | 4190.9 | 7885.2 | 12236.4 | 24437.1 |
| P50 | 255.9 | 295.8 | 313.3 | 554.9 | 711.1 | 945.0 | 1410.4 | 1874.4 | 2473.7 | 3603.5 |
| P90 | 453.6 | 543.4 | 589.2 | 1119.9 | 1493.7 | 2064.1 | 4583.2 | 14332.5 | 45042.0 | 106156.8 |
| P95 | 520.7 | 619.7 | 687.3 | 1358.5 | 1910.9 | 3941.5 | 27324.9 | 55763.3 | 81136.7 | 139534.3 |
| P99 | 612.4 | 822.1 | 1007.6 | 3005.8 | 13652.0 | 27193.5 | 54954.2 | 83933.3 | 115867.3 | 178139.4 |
| P999 | 733.0 | 1217.7 | 3214.9 | 7638.0 | 14406.9 | 30214.3 | 61711.1 | 93520.8 | 132293.4 | 191547.7 |
| Maximum | 1142.3 | 1654.7 | 3289.8 | 7641.8 | 15308.3 | 30254.7 | 61722.0 | 93532.6 | 132310.6 | 191557.9 |
| Time Per Output Token (TPOT) (ms) | ||||||||||
| Minimum | 26.2 | 30.6 | 33.3 | 53.3 | 60.0 | 66.1 | 110.7 | 109.8 | 163.2 | 289.3 |
| Average | 29.1 | 41.4 | 59.2 | 117.6 | 188.6 | 332.1 | 601.0 | 861.6 | 1068.9 | 1514.7 |
| P50 | 28.8 | 40.9 | 58.6 | 116.5 | 181.6 | 305.9 | 566.4 | 711.8 | 1017.9 | 1437.8 |
| P90 | 31.7 | 46.5 | 68.3 | 141.4 | 251.7 | 489.6 | 925.2 | 1881.4 | 1656.1 | 2448.4 |
| P95 | 32.5 | 48.4 | 71.0 | 149.1 | 278.9 | 547.6 | 1040.5 | 2048.2 | 1977.3 | 2616.5 |
| P99 | 33.9 | 53.1 | 77.7 | 161.9 | 335.7 | 615.5 | 1275.3 | 2300.8 | 2290.2 | 3033.7 |
| P999 | 35.4 | 59.7 | 91.4 | 179.4 | 372.6 | 687.0 | 1297.6 | 2348.3 | 2335.5 | 3286.4 |
| Maximum | 36.5 | 63.8 | 98.4 | 190.8 | 393.4 | 696.5 | 1301.4 | 2363.3 | 2423.9 | 3373.4 |
| Query Latency (ms) | ||||||||||
| Minimum | 3434.2 | 4046.0 | 4451.8 | 7104.2 | 8361.7 | 9827.3 | 14742.3 | 14452.1 | 22115.4 | 38962.8 |
| Average | 3977.5 | 5589.6 | 7884.9 | 15638.0 | 25000.3 | 44000.2 | 80523.0 | 117302.4 | 147992.8 | 216804.5 |
| P50 | 3952.8 | 5532.0 | 7806.5 | 15472.2 | 24073.8 | 40781.7 | 74850.6 | 94636.2 | 136765.5 | 192964.3 |
| P90 | 4358.9 | 6271.2 | 9067.9 | 18769.5 | 33068.6 | 65139.6 | 129268.5 | 245693.1 | 237623.2 | 355790.0 |
| P95 | 4474.7 | 6530.1 | 9440.4 | 19734.6 | 36730.9 | 70851.1 | 140274.1 | 265098.1 | 275355.0 | 418511.6 |
| P99 | 4689.5 | 7158.7 | 10288.6 | 21581.0 | 43729.9 | 79373.4 | 164225.2 | 296386.9 | 326615.5 | 467813.3 |
| P999 | 4930.6 | 7953.6 | 12029.7 | 23503.1 | 48166.6 | 88689.2 | 167302.5 | 301168.8 | 347126.8 | 479231.4 |
| Maximum | 5173.9 | 8470.5 | 13117.3 | 25038.5 | 50329.7 | 89355.8 | 168831.7 | 302755.7 | 347140.4 | 479234.6 |