Submission Info

Submitter Krai

Submission ID bdeb8843-219d-4adc-aef6-08b7b773cf63

Submission Status PUBLISHED

Submission Date 2026-06-25

Publication Date —

Division Standardized

Model

Model Name Llama 3 8B

Division Standardized

Model Precision FP16

Model Link Hugging Face

Calibration Link GitHub

Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.

Datasets

Dataset Name CNN-Daily Mail

Dataset Type text

Average Input Tokens 800

Dataset Link GitHub

System TPS vs. Tokens/s per User

System TPS vs. Concurrency

Time to First Token P95 vs. Concurrency

Tokens/s per User vs. Concurrency

SUT Summary

System Name Dell PowerEdge XE9680

System Availability Available

System Category Datacenter · —

System Size 8x MI300X-192GB

Node Types 1

Total Node Count 1

Max Supported Concurrency 2,048

Node Type ID 0

Node Count 1

Software

Framework vLLM v0.19.1

Serving Framework vLLM v0.19.1

Inference Backend ROCm

Driver ROCm Driver 6.10.5

Link to Container https://hub.docker.com/layers/rocm/vllm/rocm7.13.0_gfx94X-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1

Operating System ubuntu 22.04

Filesystem ext4 ramfs vfat xfs

Other Software ROCm Driver 6.10.5

Software Notes Powered by the KRAI X automation technology

Hardware

Processor

Processor Model Name Intel(R) Xeon(R) Platinum 8460Y+

Processors per Node 2

Cores Per Processor 80

VCPUs Per Processor 160

Accelerator

Accelerator Model Name AMD Instinct MI300X-NPS1-SPX-192GB-750W

Accelerators per Node 8

Memory Type HBM3

Memory Capacity 192GB

Accelerator Interconnect XGMI

Host-Accelerator Interconnect PCIe Gen5 x16

Host / Storage

Host Memory Capacity 2.1T

Memory Configuration 32 x 64GB DDR5-4800

Storage Capacity 3.1 GB NVMe SSD, 12.1 TB SSD

Storage Type NVMe SSD

Networking

Network Interfaces 2x Ethernet

Network Details Ethernet

Network Topology —

Other

Other Hardware —

Cooling Air-cooled

Hardware Notes —

Run Data

Field Name	Run 1	Run 2	Run 3	Run 4	Run 5	Run 6	Run 7	Run 8
Run Date	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026
Concurrency	32	40	64	128	256	512	1,024	2,048
System Tokens/Second	4,565.0	5,232.7	7,362.6	11,314.9	15,691.3	19,519.8	22,250.2	23,364.2
Tokens/Second per User	142.7	130.8	115.0	88.4	61.3	38.1	21.7	11.4
TTFT P95 (ms)	160.0	172.5	265.8	448.6	847.8	1338.7	1438.9	1384.0
Utilization	19.5%	22.4%	31.5%	48.4%	67.2%	83.5%	95.2%	100.0%
Configuration Summary	low_latency · c32	low_throughput · c40	med_throughput · c64	med_throughput · c128	high_throughput · c256	high_throughput · c512	high_throughput · c1024	high_throughput · c2048
Config Type	—	—	—	—	—	—	—	—
Config Region	low_latency	low_throughput	med_throughput	med_throughput	high_throughput	high_throughput	high_throughput	high_throughput
Target QPS	—	—	—	—	—	—	—	—
Result State	complete	complete	complete	complete	complete	complete	complete	complete
Measured Accuracy Score	—	—	—	—	—	—	—	—
Benchmark Version	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a
Result Version	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0
Archive URI	gs://mlperf-endpoints-blob/runs/abaa8812-4c72-4ed0-aa3c-12283480021e/archive.tar.gz	gs://mlperf-endpoints-blob/runs/b2d36e2d-aaaf-447e-be42-0ff334874e43/archive.tar.gz	gs://mlperf-endpoints-blob/runs/aaa12ccf-7bcd-48f5-8510-6c8cb12a90fc/archive.tar.gz	gs://mlperf-endpoints-blob/runs/c3a84b06-19d5-46dd-8f7f-f37b8a752e31/archive.tar.gz	gs://mlperf-endpoints-blob/runs/eeb3af35-6695-440f-a154-3ce33beb9e97/archive.tar.gz	gs://mlperf-endpoints-blob/runs/dcf96ca4-8835-4225-abff-031628ecdb8c/archive.tar.gz	gs://mlperf-endpoints-blob/runs/92715863-afef-41d7-893e-ecd32da4b758/archive.tar.gz	gs://mlperf-endpoints-blob/runs/c174a8c2-13bc-4dbc-803f-c663355220b5/archive.tar.gz
QPS	35.6644	40.8804	57.5206	88.3977	122.5887	152.4987	173.8295	182.5329
Total Output Tokens	15,399,925	15,399,918	15,399,919	15,399,920	15,399,918	15,399,924	15,399,926	15,399,921
Average Output Tokens	128	128	128	128	128	128	128	128
Run Duration (s)	3,373.45	2,943.02	2,091.63	1,361.03	981.43	788.94	692.13	659.12
Total Queries	120,312	120,312	120,312	120,312	120,312	120,312	120,312	120,312
Total Queries Issued	120,312	120,312	120,312	120,312	120,312	120,312	120,312	120,312
Total Queries Failed	0	0	0	0	0	0	0	0
Time to First Token (TTFT) P95 (ms)
Minimum	25.8	25.6	20.0	24.2	32.6	39.9	53.9	86.4
Average	86.3	87.1	121.3	235.1	447.6	730.8	883.1	909.7
P50	75.4	71.8	93.4	245.5	445.3	750.1	957.7	836.7
P90	143.7	153.2	236.6	404.3	780.1	1229.9	1321.5	1260.5
P95	160.0	172.5	265.8	448.6	847.8	1338.7	1438.9	1384.0
P99	190.3	204.9	308.8	535.2	979.0	1572.1	1792.0	3265.8
P999	223.0	239.4	349.7	625.4	1094.5	1828.2	3373.0	7169.8
Maximum	302.8	307.6	489.5	841.3	1252.7	2194.5	3865.1	7868.9
Time Per Output Token (TPOT) (ms)
Minimum	5.28	5.21	5.53	5.33	6.26	8.41	9.27	12.3
Average	6.38	7.02	7.80	9.55	12.9	20.6	39.3	80.9
P50	6.38	7.05	7.89	9.48	12.8	20.2	38.8	81.2
P90	6.87	7.49	8.67	11.2	15.7	25.1	43.9	85.2
P95	7.01	7.62	8.88	11.6	16.3	26.0	45.4	86.5
P99	7.30	7.86	9.28	12.3	17.3	27.4	47.5	89.3
P999	7.66	8.15	9.74	13.1	18.3	28.9	49.4	91.7
Maximum	8.29	8.64	10.6	14.2	20.5	30.3	52.8	96.1
Query Latency (ms)
Minimum	711.3	716.8	756.5	740.0	874.9	1134.1	1248.2	2392.6
Average	897.1	978.3	1112.3	1447.3	2086.1	3352.3	5871.4	11186.5
P50	893.5	975.8	1108.5	1443.8	2085.9	3353.7	5861.6	11189.2
P90	943.0	1024.1	1186.7	1573.8	2245.7	3623.6	6286.4	11693.5
P95	959.9	1039.5	1209.7	1612.4	2297.0	3723.3	6428.6	11872.3
P99	995.7	1071.1	1257.0	1687.6	2390.9	3947.3	6772.2	13155.5
P999	1040.2	1111.5	1316.5	1772.0	2525.0	4261.0	7942.0	17137.3
Maximum	1134.7	1233.9	1425.7	1872.3	2728.1	4659.7	8681.4	18021.5