Submitter Intel
Submission ID f82a98ec-b5f8-42ea-b1c4-ded6f25a822b
Submission Status PUBLISHED
Submission Date 2026-06-26
Publication Date —
Division Standardized
Model Name Llama 3 8B
Division Standardized
Model Precision FP16
Model Link Hugging Face
Calibration Link GitHub
Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P95 vs. Concurrency
Tokens/s per User vs. Concurrency
System Name 1-node-2S-GNR_128C
System Availability Available
System Category Datacenter · —
System Size 1x2S-GNR-AP
Node Types 1
Total Node Count 1
Max Supported Concurrency 3,072
Node Type ID 0
Node Count 1
Framework vLLM 0.23.0
Serving Framework vLLM 0.23.0
Inference Backend Not detected: CUDA/ROCm/XPU runtime not found
Driver Not available
Link to Container —
Operating System centos 9
Filesystem ext4 iso9660 ramfs vfat xfs
Other Software —
Software Notes —
Processor
Processor Model Name Intel(R) Xeon(R) 6980P
Processors per Node 2
Cores Per Processor 256
VCPUs Per Processor 256
Accelerator
Accelerator Model Name Not available
Accelerators per Node 0
Memory Type Not available
Memory Capacity Not available
Accelerator Interconnect Not available
Host-Accelerator Interconnect Not available
Host / Storage
Host Memory Capacity 2.3T
Memory Configuration 2304GB (24x96GB DDR5 8800 MT/s)
Storage Capacity 4.9 TB NVMe SSD, 243 GB SSD
Storage Type NVMe SSD
Networking
Network Interfaces 1x Ethernet
Network Details Ethernet
Network Topology —
Other
Other Hardware —
Cooling —
Hardware Notes —
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 | Run 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Run Date | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 |
| Concurrency | 3 | 16 | 40 | 128 | 256 | 512 | 1,024 | 1,536 | 2,048 | 3,072 |
| System Tokens/Second | 96.5 | 366.2 | 648.8 | 1,045.1 | 1,306.1 | 1,479.3 | 1,607.4 | 1,636.3 | 1,742.2 | 1,767.3 |
| Tokens/Second per User | 32.2 | 22.9 | 16.2 | 8.2 | 5.1 | 2.9 | 1.6 | 1.1 | 0.9 | 0.6 |
| TTFT P95 (ms) | 520.7 | 619.7 | 687.3 | 1358.5 | 1910.9 | 3941.5 | 27324.9 | 55763.3 | 81136.7 | 139534.3 |
| Utilization | 5.5% | 20.7% | 36.7% | 59.1% | 73.9% | 83.7% | 91.0% | 92.6% | 98.6% | 100.0% |
| Configuration Summary | llama-3.1-8b-concurrency · submission · c3 | llama-3.1-8b-concurrency · submission · c16 | llama-3.1-8b-concurrency · submission · c40 | llama-3.1-8b-concurrency · submission · c128 | llama-3.1-8b-concurrency · submission · c256 | llama-3.1-8b-concurrency · submission · c512 | llama-3.1-8b-concurrency · submission · c1024 | llama-3.1-8b-concurrency · submission · c1536 | llama-3.1-8b-concurrency · submission · c2048 | llama-3.1-8b-concurrency · submission · c3072 |
| Config Type | submission | submission | submission | submission | submission | submission | submission | submission | submission | submission |
| Config Region | — | — | — | — | — | — | — | — | — | — |
| Target QPS | — | — | — | — | — | — | — | — | — | — |
| Result State | complete | complete | complete | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — | — | — | — |
| Benchmark Version | bbc8697 | bbc8697 | bbc8697 | bbc8697 | c81001b | c81001b | c81001b | bbc8697 | c81001b | c81001b |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/f2ed1200-7d9f-4a09-b222-0d0b6533c440/archive.tar.gz | gs://mlperf-endpoints-blob/runs/fb68bc14-e88f-4803-8d67-9ea61be1544a/archive.tar.gz | gs://mlperf-endpoints-blob/runs/802201ce-3015-4c24-8401-37b1b8025048/archive.tar.gz | gs://mlperf-endpoints-blob/runs/08cd248c-087f-486e-b173-d58df0a1ff3c/archive.tar.gz | gs://mlperf-endpoints-blob/runs/72e238f4-b53c-428d-8b53-1b6a9da0b690/archive.tar.gz | gs://mlperf-endpoints-blob/runs/4b98ac30-88a4-41e3-ba5e-c8a1abc6fd27/archive.tar.gz | gs://mlperf-endpoints-blob/runs/f8b8081e-49f6-4a84-a1eb-6cc0b084dd63/archive.tar.gz | gs://mlperf-endpoints-blob/runs/d16a14db-3524-4281-aca7-02e158498689/archive.tar.gz | gs://mlperf-endpoints-blob/runs/91dcd2dd-996a-4629-8280-245790ecfb17/archive.tar.gz | gs://mlperf-endpoints-blob/runs/dbef15d2-82c9-4546-b316-09e2b8d7acd3/archive.tar.gz |
| QPS | 0.7541 | 2.8608 | 5.0690 | 8.1645 | 10.2036 | 11.5571 | 12.5578 | 12.7835 | 13.6109 | 13.8067 |
| Total Output Tokens | 1,711,103 | 1,711,101 | 1,711,101 | 1,711,104 | 1,711,102 | 1,711,102 | 1,711,099 | 1,711,102 | 1,711,102 | 1,711,103 |
| Average Output Tokens | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
| Run Duration (s) | 17,726.31 | 4,672.83 | 2,637.19 | 1,637.33 | 1,310.12 | 1,156.69 | 1,064.52 | 1,045.72 | 982.15 | 968.22 |
| Total Queries | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 |
| Total Queries Issued | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 | 13,368 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) P95 (ms) | ||||||||||
| Minimum | 85.9 | 89.5 | 105.1 | 135.5 | 169.3 | 227.8 | 240.0 | 225.1 | 233.2 | 267.1 |
| Average | 285.8 | 337.1 | 368.4 | 702.7 | 1051.9 | 1817.2 | 4190.9 | 7885.2 | 12236.4 | 24437.1 |
| P50 | 255.9 | 295.8 | 313.3 | 554.9 | 711.1 | 945.0 | 1410.4 | 1874.4 | 2473.7 | 3603.5 |
| P90 | 453.6 | 543.4 | 589.2 | 1119.9 | 1493.7 | 2064.1 | 4583.2 | 14332.5 | 45042.0 | 106156.8 |
| P95 | 520.7 | 619.7 | 687.3 | 1358.5 | 1910.9 | 3941.5 | 27324.9 | 55763.3 | 81136.7 | 139534.3 |
| P99 | 612.4 | 822.1 | 1007.6 | 3005.8 | 13652.0 | 27193.5 | 54954.2 | 83933.3 | 115867.3 | 178139.4 |
| P999 | 733.0 | 1217.7 | 3214.9 | 7638.0 | 14406.9 | 30214.3 | 61711.1 | 93520.8 | 132293.4 | 191547.7 |
| Maximum | 1142.3 | 1654.7 | 3289.8 | 7641.8 | 15308.3 | 30254.7 | 61722.0 | 93532.6 | 132310.6 | 191557.9 |
| Time Per Output Token (TPOT) (ms) | ||||||||||
| Minimum | 26.2 | 30.6 | 33.3 | 53.3 | 60.0 | 66.1 | 110.7 | 109.8 | 163.2 | 289.3 |
| Average | 29.1 | 41.4 | 59.2 | 117.6 | 188.6 | 332.1 | 601.0 | 861.6 | 1068.9 | 1514.7 |
| P50 | 28.8 | 40.9 | 58.6 | 116.5 | 181.6 | 305.9 | 566.4 | 711.8 | 1017.9 | 1437.8 |
| P90 | 31.7 | 46.5 | 68.3 | 141.4 | 251.7 | 489.6 | 925.2 | 1881.4 | 1656.1 | 2448.4 |
| P95 | 32.5 | 48.4 | 71.0 | 149.1 | 278.9 | 547.6 | 1040.5 | 2048.2 | 1977.3 | 2616.5 |
| P99 | 33.9 | 53.1 | 77.7 | 161.9 | 335.7 | 615.5 | 1275.3 | 2300.8 | 2290.2 | 3033.7 |
| P999 | 35.4 | 59.7 | 91.4 | 179.4 | 372.6 | 687.0 | 1297.6 | 2348.3 | 2335.5 | 3286.4 |
| Maximum | 36.5 | 63.8 | 98.4 | 190.8 | 393.4 | 696.5 | 1301.4 | 2363.3 | 2423.9 | 3373.4 |
| Query Latency (ms) | ||||||||||
| Minimum | 3434.2 | 4046.0 | 4451.8 | 7104.2 | 8361.7 | 9827.3 | 14742.3 | 14452.1 | 22115.4 | 38962.8 |
| Average | 3977.5 | 5589.6 | 7884.9 | 15638.0 | 25000.3 | 44000.2 | 80523.0 | 117302.4 | 147992.8 | 216804.5 |
| P50 | 3952.8 | 5532.0 | 7806.5 | 15472.2 | 24073.8 | 40781.7 | 74850.6 | 94636.2 | 136765.5 | 192964.3 |
| P90 | 4358.9 | 6271.2 | 9067.9 | 18769.5 | 33068.6 | 65139.6 | 129268.5 | 245693.1 | 237623.2 | 355790.0 |
| P95 | 4474.7 | 6530.1 | 9440.4 | 19734.6 | 36730.9 | 70851.1 | 140274.1 | 265098.1 | 275355.0 | 418511.6 |
| P99 | 4689.5 | 7158.7 | 10288.6 | 21581.0 | 43729.9 | 79373.4 | 164225.2 | 296386.9 | 326615.5 | 467813.3 |
| P999 | 4930.6 | 7953.6 | 12029.7 | 23503.1 | 48166.6 | 88689.2 | 167302.5 | 301168.8 | 347126.8 | 479231.4 |
| Maximum | 5173.9 | 8470.5 | 13117.3 | 25038.5 | 50329.7 | 89355.8 | 168831.7 | 302755.7 | 347140.4 | 479234.6 |