Menu
Back previous/next report: k j

Krai Dell PowerEdge XE9680

Submitted by Kraion 2026-06-26

Submission Info
Submitter Krai
Submission ID d2680e81-1dbc-4001-a438-0bab4351a118
Submission Status PUBLISHED
Submission Date 2026-06-26
Publication Date —
Division Standardized
Model
Model Name Llama 3 8B
Division Standardized
Model Precision FP16
Model Link Hugging Face
Calibration Link GitHub
Model Notes W4A8 quantization with vLLM's quantization tools, with fp8 kv cache. No other model transformations applied.
Datasets
Dataset Name CNN-Daily Mail
Dataset Type text
Average Input Tokens 800
Dataset Link GitHub

System TPS vs. Tokens/s per User

5.0k10k15k20k25kSystem TPS50100Tokens/s per User

System TPS vs. Concurrency

5.0k10k15k20k25kSystem TPS05001,0001,5002,000# of queries

Time to First Token P95 vs. Concurrency

0246TTFT P95 (s)05001,0001,5002,000# of queries

Tokens/s per User vs. Concurrency

50100Tokens/s per User05001,0001,5002,000# of queries
SUT Summary
Intel
System Name Dell PowerEdge XE9680
System Availability Available
System Category Datacenter · —
System Size 8x MI300X-192GB
Node Types 1
Total Node Count 1
Max Supported Concurrency 2,048
Node Type ID 0
Node Count 1
Software
Ubuntu
Framework vLLM v0.19.1
Serving Framework vLLM v0.19.1
Inference Backend ROCm
Driver ROCm Driver 6.10.5
Link to Container https://hub.docker.com/layers/rocm/vllm/rocm7.13.0_gfx94X-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1
Operating System ubuntu 22.04
Filesystem ext4 ramfs vfat xfs
Other Software ROCm Driver 6.10.5
Software Notes Powered by the KRAI X automation technology
Hardware

Processor

Processor Model Name Intel(R) Xeon(R) Platinum 8460Y+
Processors per Node 2
Cores Per Processor 80
VCPUs Per Processor 160

Accelerator

Accelerator Model Name AMD Instinct MI300X-NPS1-SPX-192GB-750W
Accelerators per Node 8
Memory Type HBM3
Memory Capacity 192GB
Accelerator Interconnect XGMI
Host-Accelerator Interconnect PCIe Gen5 x16

Host / Storage

Host Memory Capacity 2.1T
Memory Configuration 32 x 64GB DDR5-4800
Storage Capacity 3.1 GB NVMe SSD, 12.1 TB SSD
Storage Type NVMe SSD

Networking

Network Interfaces 2x Ethernet
Network Details Ethernet
Network Topology —

Other

Other Hardware —
Cooling Air-cooled
Hardware Notes —
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8
Run Date06/26/202606/26/202606/26/202606/26/202606/26/202606/26/202606/26/202606/26/2026
Concurrency3240641282565121,0242,048
System Tokens/Second4,401.95,078.17,106.611,442.016,362.321,290.225,082.627,268.5
Tokens/Second per User137.6127.0111.089.463.941.624.513.3
TTFT P95 (ms)157.2166.0225.8503.9939.91796.93325.76354.4
Utilization16.1%18.6%26.1%42.0%60.0%78.1%92.0%100.0%
Configuration Summaryllama3_1_8b-benchmark · online · c32llama3_1_8b-benchmark · online · c40llama3_1_8b-benchmark · online · c64llama3_1_8b-benchmark · online · c128llama3_1_8b-benchmark · online · c256llama3_1_8b-benchmark · online · c512llama3_1_8b-benchmark · online · c1024llama3_1_8b-benchmark · online · c2048
Config Typeonlineonlineonlineonlineonlineonlineonlineonline
Config Region————————
Target QPS————————
Result Statecompletecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score————————
Benchmark Versiondad7d1adad7d1adad7d1adad7d1adad7d1adad7d1adad7d1adad7d1a
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/7c71d0df-5161-4783-a5df-85e1d133ccdd/archive.tar.gzgs://mlperf-endpoints-blob/runs/0ff21ffd-691f-426e-918d-4d9d1071eb54/archive.tar.gzgs://mlperf-endpoints-blob/runs/abd3c07a-f225-40ed-9564-4f508bb47f67/archive.tar.gzgs://mlperf-endpoints-blob/runs/0524d1f0-e10a-4138-9a39-17b383f3c70b/archive.tar.gzgs://mlperf-endpoints-blob/runs/d84fb2c8-75cd-4a6b-958d-dfa8c9bd6fb1/archive.tar.gzgs://mlperf-endpoints-blob/runs/d9ee6ede-c0a0-459b-9e75-d8fb5895ed81/archive.tar.gzgs://mlperf-endpoints-blob/runs/af087ba5-a0e4-4952-9bc9-f17dffe21f47/archive.tar.gzgs://mlperf-endpoints-blob/runs/4e0753bc-22fb-49f0-934e-6c7b9ba16224/archive.tar.gz
QPS34.390239.672455.520189.3910127.8304166.3300195.9576213.0356
Total Output Tokens17,111,02917,111,03017,111,02317,111,03117,111,02717,111,02817,111,03017,111,029
Average Output Tokens128128128128128128128128
Run Duration (s)3,887.163,369.602,407.781,495.451,045.76803.70682.19627.50
Total Queries133,680133,680133,680133,680133,680133,680133,680133,680
Total Queries Issued133,680133,680133,680133,680133,680133,680133,680133,680
Total Queries Failed00000000
Time to First Token (TTFT) P95 (ms)
Minimum18.820.921.524.228.436.746.580.2
Average82.583.397.0283.5619.21223.32225.23805.6
P5070.068.374.5314.5728.41468.62481.24097.9
P90140.1147.1192.0468.0898.81732.63209.46167.1
P95157.2166.0225.8503.9939.91796.93325.76354.4
P99188.0200.2282.5565.81021.31906.03499.06648.5
P999221.0236.5338.5638.71118.32049.93655.56957.3
Maximum264.4274.5403.2696.31217.22127.93719.77135.4
Time Per Output Token (TPOT) (ms)
Minimum5.555.635.916.357.157.527.4117.2
Average6.687.288.319.0410.914.623.445.2
P506.687.298.378.689.9112.421.343.7
P907.187.829.0610.914.721.835.565.8
P957.327.969.2611.415.323.038.370.4
P997.608.259.6412.116.324.440.974.4
P9997.928.5510.112.817.325.742.876.8
Maximum8.408.9510.515.018.527.644.779.8
Query Latency (ms)
Minimum753.8812.2816.6885.31044.01026.01021.72605.4
Average930.31008.01152.51431.32000.93073.15202.39541.4
P50926.91005.61149.71430.72001.63079.65214.39577.9
P90983.01066.11229.01549.62150.23275.15489.010001.9
P951000.61085.11253.81586.62192.13335.75561.410143.4
P991036.41121.31302.61661.92284.33445.65711.610665.1
P9991077.21162.31356.81747.12410.13565.05990.411592.2
Maximum1149.31217.61467.21961.82461.93630.16290.811829.1