Submitter Intel
Submission ID 482a9d35-4e93-4643-b6af-d5dac03316cd
Submission Status PUBLISHED
Submission Date 2026-06-26
Publication Date —
Division Standardized
Model Name Llama 3 8B
Division Standardized
Model Precision FP16
Model Link Hugging Face
Calibration Link GitHub
Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.
System TPS vs. Tokens/s per User
System TPS vs. Concurrency
Time to First Token P95 vs. Concurrency
Tokens/s per User vs. Concurrency
System Name 4x Arc(TM) Pro B70
System Availability Available
System Category Datacenter · —
System Size 4x Intel(R) Arc(TM) Pro B70 Graphics
Node Types 1
Total Node Count 1
Max Supported Concurrency 1,536
Node Type ID 0
Node Count 1
Framework vLLM 0.1.dev11435+gfa77e1a63
Serving Framework vLLM 0.1.dev11435+gfa77e1a63
Inference Backend Not detected: CUDA/ROCm/XPU runtime not found
Driver Not available
Link to Container —
Operating System linux debian ubuntu 24.04
Filesystem ext4 vfat
Other Software —
Software Notes —
Processor
Processor Model Name Intel(R) Xeon(R) 698X
Processors per Node 1
Cores Per Processor 86
VCPUs Per Processor 172
Accelerator
Accelerator Model Name Intel(R) Arc(TM) Pro B70 Graphics
Accelerators per Node 4
Memory Type GDDR6
Memory Capacity 32GiB
Accelerator Interconnect Not available
Host-Accelerator Interconnect PCIe 5.0 x16
Host / Storage
Host Memory Capacity 134G
Memory Configuration 8 x 16GB DDR5-6400
Storage Capacity 1.8 TB NVMe SSD
Storage Type NVMe SSD
Networking
Network Interfaces 3x Ethernet
Network Details Ethernet
Network Topology —
Other
Other Hardware —
Cooling —
Hardware Notes —
| Field Name | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | Run 6 | Run 7 | Run 8 | Run 9 |
|---|---|---|---|---|---|---|---|---|---|
| Run Date | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 | 06/26/2026 |
| Concurrency | 16 | 32 | 40 | 64 | 128 | 256 | 512 | 1,024 | 1,536 |
| System Tokens/Second | 693.1 | 1,281.3 | 1,570.2 | 2,228.3 | 3,446.5 | 4,248.8 | 4,549.4 | 4,613.2 | 4,612.3 |
| Tokens/Second per User | 43.3 | 40.0 | 39.3 | 34.8 | 26.9 | 16.6 | 8.9 | 4.5 | 3.0 |
| TTFT P95 (ms) | 192.1 | 213.7 | 222.7 | 249.9 | 289.8 | 433.9 | 773.3 | 5946.0 | 19687.3 |
| Utilization | 15.0% | 27.8% | 34.0% | 48.3% | 74.7% | 92.1% | 98.6% | 100.0% | 100.0% |
| Configuration Summary | llama-3.1-8b-concurrency · submission · c16 | llama-3.1-8b-concurrency · submission · c32 | llama-3.1-8b-concurrency · submission · c40 | llama-3.1-8b-concurrency · submission · c64 | llama-3.1-8b-concurrency · submission · c128 | llama-3.1-8b-concurrency · submission · c256 | llama-3.1-8b-concurrency · submission · c512 | llama-3.1-8b-concurrency · submission · c1024 | llama-3.1-8b-concurrency · submission · c1536 |
| Config Type | submission | submission | submission | submission | submission | submission | submission | submission | submission |
| Config Region | — | — | — | — | — | — | — | — | — |
| Target QPS | 35.0000 | 35.0000 | 35.0000 | 35.0000 | 35.0000 | 35.0000 | 35.0000 | 35.0000 | 35.0000 |
| Result State | complete | complete | complete | complete | complete | complete | complete | complete | complete |
| Measured Accuracy Score | — | — | — | — | — | — | — | — | — |
| Benchmark Version | 7251d70 | 7251d70 | 7251d70 | 7251d70 | 7251d70 | 7251d70 | 7251d70 | 7251d70 | 7251d70 |
| Result Version | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 | 0.1.0 |
| Archive URI | gs://mlperf-endpoints-blob/runs/6b8a596c-c7b0-4aa4-b3bb-5e21ecf513e5/archive.tar.gz | gs://mlperf-endpoints-blob/runs/15f2b51a-7273-4481-8832-afe6c9679aae/archive.tar.gz | gs://mlperf-endpoints-blob/runs/1f9661d8-b56d-4f3a-b590-3a2a42eae969/archive.tar.gz | gs://mlperf-endpoints-blob/runs/5d828820-2e60-4025-a443-0a22b2f1694d/archive.tar.gz | gs://mlperf-endpoints-blob/runs/14fafa04-daf1-4b13-b003-71291f5310ce/archive.tar.gz | gs://mlperf-endpoints-blob/runs/f14bf3bb-3659-4e2d-aa16-6e5712310d46/archive.tar.gz | gs://mlperf-endpoints-blob/runs/5b8cc8de-8990-4db3-9300-ad974d418f8b/archive.tar.gz | gs://mlperf-endpoints-blob/runs/074db4fd-49d0-43ed-99f1-b50a92b4718c/archive.tar.gz | gs://mlperf-endpoints-blob/runs/f5d6ff9f-c7dc-4f65-98c1-859a5e2ef09c/archive.tar.gz |
| QPS | 5.4149 | 10.0103 | 12.2672 | 17.4090 | 26.9256 | 33.1940 | 35.5420 | 36.0407 | 36.0337 |
| Total Output Tokens | 1,711,102 | 1,711,102 | 1,711,102 | 1,711,102 | 3,422,204 | 3,422,204 | 3,422,204 | 3,422,204 | 3,422,204 |
| Average Output Tokens | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 | 128 |
| Run Duration (s) | 2,468.73 | 1,335.42 | 1,089.74 | 767.88 | 992.96 | 805.45 | 752.24 | 741.83 | 741.97 |
| Total Queries | 13,368 | 13,368 | 13,368 | 13,368 | 26,736 | 26,736 | 26,736 | 26,736 | 26,736 |
| Total Queries Issued | 13,368 | 13,368 | 13,368 | 13,368 | 26,736 | 26,736 | 26,736 | 26,736 | 26,736 |
| Total Queries Failed | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Time to First Token (TTFT) P95 (ms) | |||||||||
| Minimum | 51.0 | 52.1 | 51.7 | 51.8 | 55.5 | 56.0 | 68.3 | 119.7 | 118.1 |
| Average | 114.7 | 121.0 | 122.3 | 136.0 | 160.4 | 251.3 | 473.2 | 2449.1 | 13772.5 |
| P50 | 106.9 | 110.9 | 111.1 | 120.4 | 135.3 | 209.5 | 350.8 | 1684.4 | 13808.5 |
| P90 | 175.7 | 183.1 | 187.4 | 209.8 | 247.4 | 367.8 | 630.8 | 5124.1 | 19041.5 |
| P95 | 192.1 | 213.7 | 222.7 | 249.9 | 289.8 | 433.9 | 773.3 | 5946.0 | 19687.3 |
| P99 | 251.7 | 280.2 | 292.0 | 340.2 | 429.5 | 784.8 | 4918.5 | 18034.8 | 28677.5 |
| P999 | 333.4 | 534.1 | 745.8 | 1162.7 | 2103.1 | 4612.7 | 10059.2 | 23257.9 | 36133.6 |
| Maximum | 1219.3 | 864.0 | 1002.4 | 1525.4 | 2652.7 | 5404.0 | 11239.8 | 25102.5 | 38841.1 |
| Time Per Output Token (TPOT) (ms) | |||||||||
| Minimum | 20.2 | 20.4 | 20.4 | 20.8 | 22.1 | 22.5 | 25.1 | 30.4 | 38.4 |
| Average | 22.3 | 24.2 | 24.7 | 27.8 | 36.1 | 58.6 | 109.4 | 203.3 | 222.5 |
| P50 | 22.3 | 24.1 | 24.5 | 27.7 | 35.9 | 57.3 | 104.0 | 224.7 | 226.2 |
| P90 | 23.5 | 25.8 | 26.5 | 30.2 | 40.5 | 73.0 | 149.1 | 229.5 | 229.8 |
| P95 | 23.9 | 26.4 | 27.1 | 31.0 | 41.9 | 77.5 | 164.9 | 230.0 | 230.4 |
| P99 | 24.7 | 27.4 | 28.3 | 32.5 | 44.8 | 86.8 | 183.1 | 230.9 | 231.2 |
| P999 | 26.2 | 28.7 | 29.7 | 34.0 | 47.6 | 93.5 | 197.6 | 231.9 | 231.6 |
| Maximum | 35.7 | 29.6 | 30.8 | 36.4 | 50.0 | 97.6 | 200.6 | 232.4 | 232.5 |
| Query Latency (ms) | |||||||||
| Minimum | 2627.6 | 2665.0 | 2689.6 | 2787.3 | 2977.4 | 2964.9 | 3491.5 | 4337.1 | 11227.7 |
| Average | 2953.1 | 3193.7 | 3255.9 | 3668.9 | 4746.5 | 7695.1 | 14370.0 | 28269.7 | 42030.5 |
| P50 | 2942.8 | 3184.1 | 3237.9 | 3654.1 | 4707.9 | 7520.3 | 13644.3 | 30227.1 | 42515.9 |
| P90 | 3105.8 | 3414.1 | 3500.1 | 3985.9 | 5324.1 | 9586.4 | 19728.4 | 33711.3 | 47536.2 |
| P95 | 3162.2 | 3489.2 | 3585.2 | 4100.0 | 5518.0 | 10168.3 | 21771.0 | 34600.8 | 48150.4 |
| P99 | 3279.1 | 3619.4 | 3753.4 | 4317.1 | 5963.4 | 11442.6 | 24018.7 | 43136.2 | 56803.3 |
| P999 | 3492.3 | 3810.8 | 3982.0 | 4681.3 | 6629.7 | 12275.0 | 26147.8 | 50903.3 | 64281.4 |
| Maximum | 4676.8 | 3893.7 | 4313.8 | 5106.8 | 7393.4 | 12694.0 | 26872.5 | 53271.3 | 66466.7 |