Menu
Back previous/next report: k j

Krai Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, stock vLLM)

Submitted by Kraion 2026-10-27 Published on 2026-10-27

Result statement

Krai ran deepseek-v4_1-flash on Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, stock vLLM) and achieved a provisional 3,621.5 tokens/second System TPS at 43,128.0ms TTFT P90 in the Standardized division of the Available category on October 27, 2026. Pending MLCommons Association verification, results are subject to change pending peer review. .

[1] MLPerf Endpoints v1.0 deepseek-v4_1-flash, published October 27, 2026; retrieved from endpoints.mlcommons.org on November 7, 2026. Result —, run 7: concurrency = 512 queries, TTFT P90 = 43,128.0ms, System TPS = 3,621.5 tokens/sec, interactivity = 7.1 tokens/sec per user and total provisioned system power = 9.9 kW. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info
Krai
758a50bc-aecb-4291-ac15-fe7bc8609675
Submission Status REVIEW_PENDING
2026-10-27
2026-10-27
Division Standardized
Deployment Client on Premises
Model
Model Name deepseek-v4_1-flash
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Datasets
Dataset Name agentic_combined
Dataset Type performance
Average Input Tokens —
Dataset Link —

System TPS vs. Tokens/s per User

2.0k2.5k3.0k3.5kSystem TPS1020304050Tokens/s per User

System TPS vs. Concurrency

2.0k2.5k3.0k3.5kSystem TPS0100200300400500# of queries

Time to First Token P90 vs. Concurrency

010k20k30k40kTTFT P90 (ms)0100200300400500# of queries

Tokens/s per User vs. Concurrency

1020304050Tokens/s per User0100200300400500# of queries
SUT Summary
Intel
Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, stock vLLM)
Available
System Category Datacenter · —
8x MI300X-OAM-192GB
1
1
Max Supported Concurrency 512
Node Type ID 0
Node Count 1
Software
Ubuntu
Framework vLLM v0.31.1rc1 (public ROCm nightly build 5555a2d8f, ROCm 10.0.0) + AITER 0.1.24.post1 as shipped in the image; no patches
vLLM v0.31.1rc1 (public ROCm nightly build 5555a2d8f, ROCm 10.0.0) + AITER 0.1.24.post1 as shipped in the image; no patches
ROCm 10.0.0 in the container (HIP 7.15.26333), PyTorch 2.12.0+rocm10.0.0, Triton 3.8.0, AITER 0.1.24.post1 (MoE backend), hipBLASLt; all as shipped in the public image
amdgpu 6.10.5 (host); container ROCm 10.0.0
Link to Container vllm/vllm-openai-rocm:nightly-rocm100-5555a2d8f7116b2b4877f66516f4128176cf722a, pulled by digest sha256:e7d4d8643501172acc5e9a6fcc3739680359613c2904ec977cde9e721e798e82 (https://hub.docker.com/r/vllm/vllm-openai-rocm); used unmodified
Operating System Ubuntu 22.04.5 LTS (kernel 5.15.0-130-generic)
Filesystem XFS on LVM (/mnt/llm_data)
Other Software Docker 29.2.1, MLPerf Endpoints client d3c57f2 run from a source checkout (uv), nginx 1.27.5 (nginx:1.27.5-alpine) as the session router
Software Notes Stock software: the public vLLM ROCm nightly image used as published, with InferenceX's validated MI300X server recipe for this model (AITER MoE, DSpark k=5, Engram tables in pinned host memory, decode-only HIP graphs, bounded SWA replay off). No kernel, framework or client patches.
Hardware

Processor

Intel(R) Xeon(R) Platinum 8460Y+
2
40
VCPUs Per Processor 80

Accelerator

—
0
—
—
—
—

Host / Storage

2 TiB
Memory Configuration 2 TiB DDR5 RDIMM (2015 GiB reported by the OS)
Storage Capacity 7x 1.6 TB NVMe visible to the OS; 11 TB XFS logical volume for models and results
Storage Type Dell Ent NVMe PM1735a MU 1.6TB

Networking

2x Broadcom BCM5720 1GbE (one link up); no InfiniBand. Client and server share the node (host networking).
Ethernet (management only; benchmark traffic is in-host)
Network Topology —

Other

Other Hardware —
Cooling Air-cooled
Hardware Notes GPUs at the default 750 W power limit (rocm-smi: Max Graphics Package Power 750 W); BIOS 2.6.5 (2025-07-11); GPU VBIOS 113-M3000100-103.
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7
Run Date10/27/202610/27/202610/27/202610/27/202610/27/202610/27/202610/27/2026
Concurrency324064128256384512
System Tokens/Second1,695.41,889.32,282.22,879.03,341.13,567.33,621.5
Tokens/Second per User53.047.235.722.513.19.37.1
TTFT P90 (ms)903.8938.51263.21946.23575.07168.743128.0
Utilization46.8%52.2%63.0%79.5%92.3%98.5%100.0%
Configuration Summarydeepseek-v4.1-flash-agentic-benchmark · online · c32deepseek-v4.1-flash-agentic-benchmark · online · c40deepseek-v4.1-flash-agentic-benchmark · online · c64deepseek-v4.1-flash-agentic-benchmark · online · c128deepseek-v4.1-flash-agentic-benchmark · online · c256deepseek-v4.1-flash-agentic-benchmark · online · c384deepseek-v4.1-flash-agentic-benchmark · online · c512
Config Typeonlineonlineonlineonlineonlineonlineonline
Config Region———————
Target QPS———————
Result Statecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score———————
Benchmark Versiond3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/a3f2c1d9-4e4a-4f7e-8f60-b7b57e32e04d/archive.tar.gzgs://mlperf-endpoints-blob/runs/4a311425-54f7-495b-bbb2-9affd1fc06a5/archive.tar.gzgs://mlperf-endpoints-blob/runs/82bd76ca-0f03-4571-8118-19ed7e09fd9c/archive.tar.gzgs://mlperf-endpoints-blob/runs/64e46822-162b-43bf-ba39-4b6cd01d1e71/archive.tar.gzgs://mlperf-endpoints-blob/runs/5f889c19-fb5e-4264-8114-2cdec3c7f5b4/archive.tar.gzgs://mlperf-endpoints-blob/runs/18abb3cf-df13-432b-9e59-cffc022beb7e/archive.tar.gzgs://mlperf-endpoints-blob/runs/edc99044-5b6f-4069-aed6-51a17a34482d/archive.tar.gz
QPS2.01802.21672.69663.40463.95284.20764.1388
Total Output Tokens15,753,32415,732,70615,138,15931,200,26145,017,86658,468,37671,023,387
Average Output Tokens840.13852.31846.32845.63845.25847.82875.02
Run Duration (s)9,291.948,327.286,633.2710,837.0513,473.8916,390.0919,611.54
Total Queries18,75118,45917,88736,89653,26068,96381,168
Total Queries Issued18,75118,45917,88736,89653,26068,96381,168
Total Queries Failed0000000
Time to First Token (TTFT) (ms)
Minimum184.4183.2187.2206.4232.4433.7252.0
Average590.6609.0802.71224.72521.35211.531789.8
P50293.8303.7346.1461.0724.21651.927878.6
P90903.8938.51263.21946.23575.07168.743128.0
P952108.52226.02972.33430.64311.89494.247923.4
P993251.73392.94205.16706.517626.737038.686778.5
P99930117.631148.955472.0114677.8262756.8411402.8540463.8
Maximum59214.455170.989188.7169776.5337083.7508078.7649964.4
Time Per Output Token (TPOT) (ms)
Minimum-0.01-0.01-0.01-0.02-1.95-0.02-0.03
Average13.615.520.933.858.680.681.9
P5012.814.619.632.157.181.183.4
P9020.222.731.150.785.8115.3116.2
P9523.326.136.758.596.6128.8128.3
P9941.544.956.784.3138.0174.1166.8
P99981.083.5138.2192.9223.2227.7219.4
Maximum250.3189.7210.3372.7579.7321.0332.5
Query Latency (ms)
Minimum382.6431.5486.5694.7980.71574.48323.3
Average14089.916189.821579.235281.261532.586829.6117920.7
P502609.23045.14112.66264.210595.616194.046120.4
P9033724.737960.450248.883059.0146936.7213596.7255808.6
P9584313.599327.1130415.1226774.3400527.4555146.5687787.6
P99165779.0184940.1246360.7397534.3683391.0924690.5968466.7
P999194111.1212554.3278367.3464249.9779911.91064190.51089143.6
Maximum222594.9231189.2322515.6590364.81026646.61529591.21534737.8