Menu
Back previous/next report: k j

Krai Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, vLLM + Krai MoE kernels)

Submitted by Kraion 2026-10-27 Published on 2026-10-27

Result statement

Krai ran deepseek-v4_1-flash on Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, vLLM + Krai MoE kernels) and achieved a provisional 5,489.4 tokens/second System TPS at 1,527.6ms TTFT P90 in the Standardized division of the Available category on October 27, 2026. Pending MLCommons Association verification, results are subject to change pending peer review. .

[1] MLPerf Endpoints v1.0 deepseek-v4_1-flash, published October 27, 2026; retrieved from endpoints.mlcommons.org on November 7, 2026. Result —, run 6: concurrency = 384 queries, TTFT P90 = 1,527.6ms, System TPS = 5,489.4 tokens/sec, interactivity = 14.3 tokens/sec per user and total provisioned system power = 9.9 kW. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Submission Info
Krai
73c4b6bc-bf7b-46f3-b3ac-0acd2964cd55
Submission Status REVIEW_PENDING
2026-10-27
2026-10-27
Division Standardized
Deployment Client on Premises
Model
Model Name deepseek-v4_1-flash
Division Standardized
Model Precision —
Model Link —
Calibration Link —
Model Notes —
Datasets
Dataset Name agentic_combined
Dataset Type performance
Average Input Tokens —
Dataset Link —

System TPS vs. Tokens/s per User

3.0k4.0k5.0kSystem TPS204060Tokens/s per User

System TPS vs. Concurrency

3.0k4.0k5.0kSystem TPS0100200300400500# of queries

Time to First Token P90 vs. Concurrency

1.0k2.0k3.0k4.0kTTFT P90 (ms)0100200300400500# of queries

Tokens/s per User vs. Concurrency

204060Tokens/s per User0100200300400500# of queries
SUT Summary
Intel
Dell PowerEdge XE9680 (8x MI300X-OAM-192GB, vLLM + Krai MoE kernels)
Available
System Category Datacenter · —
8x MI300X-OAM-192GB
1
1
Max Supported Concurrency 512
Node Type ID 0
Node Count 1
Software
Ubuntu
Framework vLLM v0.31.1rc1 (nightly build 5555a2d8f; ROCm 10.0.0 stable release) + AITER 0.1.24.post1 with the gfx942 MoE a16w4 kernel patches in shared/src/mi300x_vllm/patches
vLLM v0.31.1rc1 (nightly build 5555a2d8f; ROCm 10.0.0 stable release) + AITER 0.1.24.post1 with the gfx942 MoE a16w4 kernel patches in shared/src/mi300x_vllm/patches
ROCm 10.0.0 stable release in the container (AMD wheels rocm==10.0.0 from https://stable.repo.amd.com/rocm/whl-next/) (HIP 7.15.26333), PyTorch 2.12.0+rocm10.0.0, Triton 3.8.0, AITER 0.1.24.post1 (MoE backend; gfx942 Gluon a16w4 MoE GEMM kernel, see shared/src), hipBLASLt
amdgpu 6.10.5 (host ROCm 6.3.2); container ROCm 10.0.0 (stable)
Link to Container Base image vllm/vllm-openai-rocm@sha256:e7d4d8643501172acc5e9a6fcc3739680359613c2904ec977cde9e721e798e82 (https://hub.docker.com/r/vllm/vllm-openai-rocm, tag nightly-rocm100-5555a2d8f7116b2b4877f66516f4128176cf722a); submission image built FROM that digest with the Dockerfile and patch overlay in shared/src/mi300x_vllm/docker (tag krai/vllm-openai-rocm-dsv41:frozen-20261023-c1fd5fa09d8f). Client https://hub.docker.com/r/wongcannan/mlperf-endpoints (v1.1@sha256:514e6360b981f2f59300a796f84585edaa4c6401dad352b062d4038dd9b32ea4).
Operating System Ubuntu 22.04.5 LTS (kernel 5.15.0-130-generic)
Filesystem XFS on LVM (/mnt/llm_data)
Other Software Docker 29.2.1, MLPerf Endpoints client d3c57f2 (wongcannan/mlperf-endpoints:v1.1), nginx 1.27.5 (nginx:1.27-alpine) as the session router at C32-C384, HAProxy 3.0.12 (haproxy:3.0.12-alpine@sha256:e3229153dd6429fc71f8a9a1f1ccbf8dab6962422d7043c81e84dd6a075e90ee) at C512
Software Notes Kernel patches (AITER gfx942 MoE a16w4 Gluon dequant kernel + tuned decode configs; vLLM MXFP4 intermediate-size padding to 128 on gfx942) are baked into the image from the pinned base digest; sources, diffs and upstream PR links in shared/src/mi300x_vllm/patches.
Hardware

Processor

Intel(R) Xeon(R) Platinum 8460Y+
2
40
VCPUs Per Processor 80

Accelerator

—
0
—
—
—
—

Host / Storage

2 TiB
Memory Configuration 2 TiB DDR5 RDIMM (2015 GiB reported by the OS)
Storage Capacity 7x 1.6 TB NVMe visible to the OS; 11 TB XFS logical volume for models and results
Storage Type Dell Ent NVMe PM1735a MU 1.6TB

Networking

2x Broadcom BCM5720 1GbE (one link up); no InfiniBand. Client and server share the node (host networking).
Ethernet (management only; benchmark traffic is in-host)
Network Topology —

Other

Other Hardware —
Cooling Air-cooled
Hardware Notes GPUs at the default 750 W power limit (rocm-smi: Max Graphics Package Power 750 W); BIOS 2.6.5 (2025-07-11); GPU VBIOS 113-M3000100-103.
Run Data
Field NameRun 1Run 2Run 3Run 4Run 5Run 6Run 7
Run Date10/27/202610/27/202610/27/202610/27/202610/27/202610/27/202610/27/2026
Concurrency324064128256384512
System Tokens/Second2,342.22,617.73,242.74,127.75,159.25,489.45,476.4
Tokens/Second per User73.265.450.732.220.214.310.7
TTFT P90 (ms)684.4691.7762.6973.01361.61527.64008.5
Utilization42.7%47.7%59.1%75.2%94.0%100.0%99.8%
Configuration Summaryendpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c32endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c40endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c64endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c128endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c256endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c384endpoint-dsv41f-agentic-inference-benchmark-mi300x8 · online · c512
Config Typeonlineonlineonlineonlineonlineonlineonline
Config Region———————
Target QPS———————
Result Statecompletecompletecompletecompletecompletecompletecomplete
Measured Accuracy Score———————
Benchmark Versiond3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2d3c57f2
Result Version0.1.00.1.00.1.00.1.00.1.00.1.00.1.0
Archive URIgs://mlperf-endpoints-blob/runs/974bbca9-3a2a-49da-8164-647b64cf2452/archive.tar.gzgs://mlperf-endpoints-blob/runs/9dd4e166-3557-4346-95a9-58b9fe61f080/archive.tar.gzgs://mlperf-endpoints-blob/runs/e1868ded-ae64-4ab9-84c1-03af2ce71961/archive.tar.gzgs://mlperf-endpoints-blob/runs/39188410-8326-47e2-8ef6-de953ac71efb/archive.tar.gzgs://mlperf-endpoints-blob/runs/d79fe7d6-b9a9-4831-b8a2-0025a7bf46f9/archive.tar.gzgs://mlperf-endpoints-blob/runs/a8dbc05a-168e-44b2-8e04-6bb5bbe564f4/archive.tar.gzgs://mlperf-endpoints-blob/runs/9d558c26-1afe-4b30-860a-488fdb06de8f/archive.tar.gz
QPS2.72393.10713.76594.66615.84906.10466.3436
Total Output Tokens16,198,69015,544,79015,416,56914,371,65028,079,57425,220,58155,959,414
Average Output Tokens859.89842.49861.07884.63882.06899.23863.3
Run Duration (s)6,915.905,938.324,754.283,481.735,442.664,594.4310,218.19
Total Queries18,83818,45117,90416,24631,83428,04764,820
Total Queries Issued18,83818,45117,90416,24631,83428,04764,820
Total Queries Failed0000000
Time to First Token (TTFT) (ms)
Minimum108.4118.1139.9205.3312.1407.1496.3
Average449.2463.0534.9741.81062.11499.12522.2
P50389.8399.9445.0566.8796.5921.81102.0
P90684.4691.7762.6973.01361.61527.64008.5
P95790.0791.8902.31190.21653.31920.39648.5
P991345.81464.81793.33244.33505.221939.627654.9
P99910240.910855.412638.017827.632678.553067.488943.2
Maximum12635.113797.515725.122765.439626.960575.2100772.7
Time Per Output Token (TPOT) (ms)
Minimum3.502.885.9610.617.423.019.3
Average9.6711.014.924.139.455.373.1
P509.2510.514.122.336.951.866.1
P9013.214.819.831.551.471.8101.5
P9514.516.221.734.856.579.7118.1
P9917.820.430.264.789.1127.9175.3
P99932.141.867.4107.4132.5167.3393.2
Maximum102.9126.7124.3245.6166.3320.91917.8
Query Latency (ms)
Minimum306.5346.1409.9665.9864.61233.31329.3
Average9988.311026.214938.624468.339866.756421.074014.6
P501920.02142.52829.44189.36733.99111.113418.4
P9023316.825580.335111.760183.096286.5139171.9174761.8
P9563922.966138.198368.1178431.4285510.0445061.1513624.5
P99113451.9124205.9166108.8260086.1419121.8572706.6799062.5
P999130205.4138784.5182201.3282175.4453851.0622475.01027529.0
Maximum157564.2170420.1209837.9335549.1506035.4723088.71293983.3