Back previous/next report: k j

Dell PowerEdge XE9680

Submitted by Kraion 2026-06-25

Result statement

Krai ran Llama 3.1 8B on Dell PowerEdge XE9680 and achieved a verified 23,364.2 tokens/second System TPS at 1,260.5ms TTFT P90 in the Standardized division of the Available category .

[1] MLPerf Endpoints v1.0 Llama 3.1 8B; retrieved from endpoints.mlcommons.org on October 22, 2026. Result —, run 8: concurrency = 2,048 queries, TTFT P90 = 1,260.5ms, System TPS = 23,364.2 tokens/sec and interactivity = 11.4 tokens/sec per user. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info

Krai

bdeb8843-219d-4adc-aef6-08b7b773cf63

Submission Status PUBLISHED

2026-06-25

—

Division Standardized

Deployment Client on Premises

Model

Model Name Llama 3.1 8B

Division Standardized

Model Precision FP16

Model Link Hugging Face

Calibration Link GitHub

Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.

Datasets

Dataset Name CNN-Daily Mail

Dataset Type text

Average Input Tokens 800

Dataset Link GitHub

System TPS vs. Tokens/s per User

System TPS vs. Concurrency

Time to First Token P90 vs. Concurrency

Tokens/s per User vs. Concurrency

SUT Summary

Dell PowerEdge XE9680

Available

System Category Datacenter · —

8x MI300X-192GB

Max Supported Concurrency 2,048

Node Type ID 0

Node Count 1

Software

Framework vLLM v0.19.1

vLLM v0.19.1

ROCm

ROCm Driver 6.10.5

Link to Container https://hub.docker.com/layers/rocm/vllm/rocm7.13.0_gfx94X-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1

Operating System ubuntu 22.04

Filesystem ext4 ramfs vfat xfs

Other Software ROCm Driver 6.10.5

Software Notes Powered by the KRAI X automation technology

Hardware

Processor

Intel(R) Xeon(R) Platinum 8460Y+

VCPUs Per Processor 160

Accelerator

AMD Instinct MI300X-NPS1-SPX-192GB-750W

HBM3

192GB

XGMI

PCIe Gen5 x16

Host / Storage

2.1T

Memory Configuration 32 x 64GB DDR5-4800

Storage Capacity 3.1 GB NVMe SSD, 12.1 TB SSD

Storage Type NVMe SSD

Networking

2x Ethernet

Ethernet

Network Topology —

Other

Other Hardware —

Cooling Air-cooled

Hardware Notes —

Run Data

Field Name	Run 1	Run 2	Run 3	Run 4	Run 5	Run 6	Run 7	Run 8
Run Date	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026	06/25/2026
Concurrency	32	40	64	128	256	512	1,024	2,048
System Tokens/Second	4,565.0	5,232.7	7,362.6	11,314.9	15,691.3	19,519.8	22,250.2	23,364.2
Tokens/Second per User	142.7	130.8	115.0	88.4	61.3	38.1	21.7	11.4
TTFT P90 (ms)	143.7	153.2	236.6	404.3	780.1	1229.9	1321.5	1260.5
Utilization	19.5%	22.4%	31.5%	48.4%	67.2%	83.5%	95.2%	100.0%
Configuration Summary	low_latency · c32	low_throughput · c40	med_throughput · c64	med_throughput · c128	high_throughput · c256	high_throughput · c512	high_throughput · c1024	high_throughput · c2048
Config Type	—	—	—	—	—	—	—	—
Config Region	low_latency	low_throughput	med_throughput	med_throughput	high_throughput	high_throughput	high_throughput	high_throughput
Target QPS	—	—	—	—	—	—	—	—
Result State	complete	complete	complete	complete	complete	complete	complete	complete
Measured Accuracy Score	—	—	—	—	—	—	—	—
Benchmark Version	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a	dad7d1a
Result Version	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0	0.1.0
Archive URI	gs://mlperf-endpoints-blob/runs/abaa8812-4c72-4ed0-aa3c-12283480021e/archive.tar.gz	gs://mlperf-endpoints-blob/runs/b2d36e2d-aaaf-447e-be42-0ff334874e43/archive.tar.gz	gs://mlperf-endpoints-blob/runs/aaa12ccf-7bcd-48f5-8510-6c8cb12a90fc/archive.tar.gz	gs://mlperf-endpoints-blob/runs/c3a84b06-19d5-46dd-8f7f-f37b8a752e31/archive.tar.gz	gs://mlperf-endpoints-blob/runs/eeb3af35-6695-440f-a154-3ce33beb9e97/archive.tar.gz	gs://mlperf-endpoints-blob/runs/dcf96ca4-8835-4225-abff-031628ecdb8c/archive.tar.gz	gs://mlperf-endpoints-blob/runs/92715863-afef-41d7-893e-ecd32da4b758/archive.tar.gz	gs://mlperf-endpoints-blob/runs/c174a8c2-13bc-4dbc-803f-c663355220b5/archive.tar.gz
QPS	35.6644	40.8804	57.5206	88.3977	122.5887	152.4987	173.8295	182.5329
Total Output Tokens	15,399,925	15,399,918	15,399,919	15,399,920	15,399,918	15,399,924	15,399,926	15,399,921
Average Output Tokens	128	128	128	128	128	128	128	128
Run Duration (s)	3,373.45	2,943.02	2,091.63	1,361.03	981.43	788.94	692.13	659.12
Total Queries	120,312	120,312	120,312	120,312	120,312	120,312	120,312	120,312
Total Queries Issued	120,312	120,312	120,312	120,312	120,312	120,312	120,312	120,312
Total Queries Failed	0	0	0	0	0	0	0	0
Time to First Token (TTFT) (ms)
Minimum	25.8	25.6	20.0	24.2	32.6	39.9	53.9	86.4
Average	86.3	87.1	121.3	235.1	447.6	730.8	883.1	909.7
P50	75.4	71.8	93.4	245.5	445.3	750.1	957.7	836.7
P90	143.7	153.2	236.6	404.3	780.1	1229.9	1321.5	1260.5
P95	160.0	172.5	265.8	448.6	847.8	1338.7	1438.9	1384.0
P99	190.3	204.9	308.8	535.2	979.0	1572.1	1792.0	3265.8
P999	223.0	239.4	349.7	625.4	1094.5	1828.2	3373.0	7169.8
Maximum	302.8	307.6	489.5	841.3	1252.7	2194.5	3865.1	7868.9
Time Per Output Token (TPOT) (ms)
Minimum	5.28	5.21	5.53	5.33	6.26	8.41	9.27	12.3
Average	6.38	7.02	7.80	9.55	12.9	20.6	39.3	80.9
P50	6.38	7.05	7.89	9.48	12.8	20.2	38.8	81.2
P90	6.87	7.49	8.67	11.2	15.7	25.1	43.9	85.2
P95	7.01	7.62	8.88	11.6	16.3	26.0	45.4	86.5
P99	7.30	7.86	9.28	12.3	17.3	27.4	47.5	89.3
P999	7.66	8.15	9.74	13.1	18.3	28.9	49.4	91.7
Maximum	8.29	8.64	10.6	14.2	20.5	30.3	52.8	96.1
Query Latency (ms)
Minimum	711.3	716.8	756.5	740.0	874.9	1134.1	1248.2	2392.6
Average	897.1	978.3	1112.3	1447.3	2086.1	3352.3	5871.4	11186.5
P50	893.5	975.8	1108.5	1443.8	2085.9	3353.7	5861.6	11189.2
P90	943.0	1024.1	1186.7	1573.8	2245.7	3623.6	6286.4	11693.5
P95	959.9	1039.5	1209.7	1612.4	2297.0	3723.3	6428.6	11872.3
P99	995.7	1071.1	1257.0	1687.6	2390.9	3947.3	6772.2	13155.5
P999	1040.2	1111.5	1316.5	1772.0	2525.0	4261.0	7942.0	17137.3
Maximum	1134.7	1233.9	1425.7	1872.3	2728.1	4659.7	8681.4	18021.5