Menu
Back previous/next report: k j

Krai Dell PowerEdge XE9680

Submitted by Kraion 2026-06-26

Result statement

Krai ran Llama 3.1 8B on Dell PowerEdge XE9680 and achieved a verified 27,268.5 tokens/second System TPS at 6,167.1ms TTFT P90 in the Standardized division of the Available category .

[1] MLPerf Endpoints v1.0 Llama 3.1 8B; retrieved from endpoints.mlcommons.org on October 22, 2026. Result —, run 8: concurrency = 2,048 queries, TTFT P90 = 6,167.1ms, System TPS = 27,268.5 tokens/sec and interactivity = 13.3 tokens/sec per user. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info
Krai
d2680e81-1dbc-4001-a438-0bab4351a118
Submission Status PUBLISHED
2026-06-26
—
Division Standardized
Deployment Client on Premises
Model
Model Name Llama 3.1 8B
Division Standardized
Model Precision FP16
Model Link Hugging Face
Calibration Link GitHub
Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.
Datasets
Dataset Name CNN-Daily Mail
Dataset Type text
Average Input Tokens 800
Dataset Link GitHub

System TPS vs. Tokens/s per User

5.0k10k15k20k25kSystem TPS50100Tokens/s per User

System TPS vs. Concurrency

5.0k10k15k20k25kSystem TPS05001,0001,5002,000# of queries

Time to First Token P90 vs. Concurrency

02.0k4.0k6.0kTTFT P90 (ms)05001,0001,5002,000# of queries

Tokens/s per User vs. Concurrency

50100Tokens/s per User05001,0001,5002,000# of queries
SUT Summary
Intel
Dell PowerEdge XE9680
Available
System Category Datacenter · —
8x MI300X-192GB
1
1
Max Supported Concurrency 2,048
Node Type ID 0
Node Count 1
Software
Ubuntu
Framework vLLM v0.19.1
vLLM v0.19.1
ROCm
ROCm Driver 6.10.5
Link to Container https://hub.docker.com/layers/rocm/vllm/rocm7.13.0_gfx94X-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1
Operating System ubuntu 22.04
Filesystem ext4 ramfs vfat xfs
Other Software ROCm Driver 6.10.5
Software Notes Powered by the KRAI X automation technology
Hardware

Processor

Intel(R) Xeon(R) Platinum 8460Y+
2
80
VCPUs Per Processor 160

Accelerator

AMD Instinct MI300X-NPS1-SPX-192GB-750W
8
HBM3
192GB
XGMI
PCIe Gen5 x16

Host / Storage

2.1T
Memory Configuration 32 x 64GB DDR5-4800
Storage Capacity 3.1 GB NVMe SSD, 12.1 TB SSD
Storage Type NVMe SSD

Networking

2x Ethernet
Ethernet
Network Topology —

Other

Other Hardware —
Cooling Air-cooled
Hardware Notes —
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8
Run Date06/26/202606/26/202606/26/202606/26/202606/26/202606/26/202606/26/202606/26/2026
Concurrency3240641282565121,0242,048
System Tokens/Second4,401.95,078.17,106.611,442.016,362.321,290.225,082.627,268.5
Tokens/Second per User137.6127.0111.089.463.941.624.513.3
TTFT P90 (ms)140.1147.1192.0468.0898.81732.63209.46167.1
Utilization16.1%18.6%26.1%42.0%60.0%78.1%92.0%100.0%
Configuration Summaryllama3_1_8b-benchmark · online · c32llama3_1_8b-benchmark · online · c40llama3_1_8b-benchmark · online · c64llama3_1_8b-benchmark · online · c128llama3_1_8b-benchmark · online · c256llama3_1_8b-benchmark · online · c512llama3_1_8b-benchmark · online · c1024llama3_1_8b-benchmark · online · c2048
Config Typeonlineonlineonlineonlineonlineonlineonlineonline
Config Region————————
Target QPS————————
Result Statecompletecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score————————
Benchmark Versiondad7d1adad7d1adad7d1adad7d1adad7d1adad7d1adad7d1adad7d1a
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/7c71d0df-5161-4783-a5df-85e1d133ccdd/archive.tar.gzgs://mlperf-endpoints-blob/runs/0ff21ffd-691f-426e-918d-4d9d1071eb54/archive.tar.gzgs://mlperf-endpoints-blob/runs/abd3c07a-f225-40ed-9564-4f508bb47f67/archive.tar.gzgs://mlperf-endpoints-blob/runs/0524d1f0-e10a-4138-9a39-17b383f3c70b/archive.tar.gzgs://mlperf-endpoints-blob/runs/d84fb2c8-75cd-4a6b-958d-dfa8c9bd6fb1/archive.tar.gzgs://mlperf-endpoints-blob/runs/d9ee6ede-c0a0-459b-9e75-d8fb5895ed81/archive.tar.gzgs://mlperf-endpoints-blob/runs/af087ba5-a0e4-4952-9bc9-f17dffe21f47/archive.tar.gzgs://mlperf-endpoints-blob/runs/4e0753bc-22fb-49f0-934e-6c7b9ba16224/archive.tar.gz
QPS34.390239.672455.520189.3910127.8304166.3300195.9576213.0356
Total Output Tokens17,111,02917,111,03017,111,02317,111,03117,111,02717,111,02817,111,03017,111,029
Average Output Tokens128128128128128128128128
Run Duration (s)3,887.163,369.602,407.781,495.451,045.76803.70682.19627.50
Total Queries133,680133,680133,680133,680133,680133,680133,680133,680
Total Queries Issued133,680133,680133,680133,680133,680133,680133,680133,680
Total Queries Failed00000000
Time to First Token (TTFT) (ms)
Minimum18.820.921.524.228.436.746.580.2
Average82.583.397.0283.5619.21223.32225.23805.6
P5070.068.374.5314.5728.41468.62481.24097.9
P90140.1147.1192.0468.0898.81732.63209.46167.1
P95157.2166.0225.8503.9939.91796.93325.76354.4
P99188.0200.2282.5565.81021.31906.03499.06648.5
P999221.0236.5338.5638.71118.32049.93655.56957.3
Maximum264.4274.5403.2696.31217.22127.93719.77135.4
Time Per Output Token (TPOT) (ms)
Minimum5.555.635.916.357.157.527.4117.2
Average6.687.288.319.0410.914.623.445.2
P506.687.298.378.689.9112.421.343.7
P907.187.829.0610.914.721.835.565.8
P957.327.969.2611.415.323.038.370.4
P997.608.259.6412.116.324.440.974.4
P9997.928.5510.112.817.325.742.876.8
Maximum8.408.9510.515.018.527.644.779.8
Query Latency (ms)
Minimum753.8812.2816.6885.31044.01026.01021.72605.4
Average930.31008.01152.51431.32000.93073.15202.39541.4
P50926.91005.61149.71430.72001.63079.65214.39577.9
P90983.01066.11229.01549.62150.23275.15489.010001.9
P951000.61085.11253.81586.62192.13335.75561.410143.4
P991036.41121.31302.61661.92284.33445.65711.610665.1
P9991077.21162.31356.81747.12410.13565.05990.411592.2
Maximum1149.31217.61467.21961.82461.93630.16290.811829.1