2 Nodes

System 4

Submitted by Submitter 4on 2026-02-27. Published on 2026-03-16

SUT Summary

System Name System 4

System Availability Available

System Category Datacenter · Cloud

System Size 8x Accelerator 4

Model

Model Name QWEN3 CODER 480B

Division Open

Model Precision FP8

Model Link —

Transformation Link —

Model Notes —

Datasets

Dataset Name OpenOrca

Dataset Type Performance

Average Input Tokens 190.362

Average Output Tokens 292.545

Dataset Link —

Measured Accuracy Score —

Throughput vs Interactivity

Throughput vs Concurrency

Time to First Token vs Concurrency

Interactivity vs Concurrency

Hardware

Processor

Processor Model Name Processor 4

Processors per Node 2

Cores Per Processor 32

VCPUs Per Processor —

Accelerator

Accelerator Model Name Accelerator 4

Accelerators per Node 8

Memory Type

Memory Capacity 256 GB

Accelerator Interconnect

Host-Accelerator Interconnect

Host / Storage

Host Memory Capacity 1TB

Memory Configuration

Storage Capacity

Storage Type

Cooling Liquid-cooled

Hardware Notes

Software

Framework vLLM

Operating System Linux

Other Software Inference Backend v1.0

Software Notes —

Run Data

Field Name	Run 1	Run 2	Run 3	Run 4	Run 5	Run 6	Run 7	Run 8
Run Date	02/25/2026	02/25/2026	02/25/2026	02/25/2026	02/25/2026	02/24/2026	02/24/2026	02/28/2026
Concurrency	0.54	1.09	2.18	4.36	8.71	17.42	34.85	69.70
System Tokens/Second	12.0	22.9	38.8	64.4	98.5	131.4	171.6	199.8
Tokens/Second per User	22.0	21.0	17.8	14.8	11.3	7.5	4.9	2.9
TTFT P99 (ms)	2736.7	3136.5	3651.6	4022.0	4986.4	12923.1	156250.0	408064.0
Utilization	6.0%	11.5%	19.4%	32.2%	49.3%	65.8%	85.9%	100.0%
Configuration Summary	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16	TP=4, PP=1, batch=256, precision=FP8, kv_cache=FP16
QPS	0.0411	0.0780	0.1326	0.2199	0.3358	0.4478	0.5863	0.6833
Total Output Tokens	7,176,538	7,214,114	7,189,678	7,198,828	7,208,951	7,211,649	7,193,097	7,186,180
Run Duration (s)	597,954.52	315,267.38	185,363.60	111,745.40	73,190.44	54,887.23	41,915.72	35,969.14
Total Requests	24,576	24,576	24,576	24,576	24,576	24,576	24,576	24,576
Time To First Token (TTFT) (ms)
Minimum	1163.2	1297.8	1511.0	1747.7	1953.1	3331.7	4600.8	6622.7
Average	1940.9	2015.4	2257.4	2587.5	3256.1	4728.4	9048.3	23312.8
P50	1802.2	1834.4	2003.0	2518.9	3109.1	4298.8	6331.7	11657.2
P90	2598.0	2633.7	2796.1	2994.6	3913.8	5061.3	7152.1	14908.6
P95	2627.7	2663.1	2848.8	3367.6	4035.8	5442.3	9498.4	15134.0
P99	2736.7	3136.5	3651.6	4022.0	4986.4	12923.1	156250.0	408064.0
P999	3543.2	3612.9	11091.1	22970.7	40832.6	81758.8	222715.0	483273.0
Maximum	3966.8	16722.5	11100.9	22986.0	43035.1	84065.5	234174.1	493677.7
Time Per Output Token (TPOT) (ms)
Minimum	635.8	660.5	696.5	727.7	772.9	881.4	1054.2	1450.0
Average	663.0	695.3	819.6	985.2	1285.4	1921.7	2920.8	4955.7
P50	661.9	694.2	818.4	983.9	1284.7	1922.4	2925.7	5002.4
P90	671.3	706.2	833.4	1004.5	1311.0	1952.7	2971.5	5117.6
P95	675.1	710.7	839.2	1012.1	1322.5	1966.4	2989.4	5153.0
P99	688.3	727.3	861.9	1040.1	1365.6	2023.1	3105.7	5294.5
P999	795.2	863.5	992.3	1172.9	1506.0	2306.1	3630.0	6425.4
Maximum	1423.2	1319.2	1562.2	1900.5	2315.8	4659.9	5767.2	10909.1
Request Latency (ms)
Minimum	2466.5	2518.6	3101.7	3327.1	3771.5	5382.7	8257.9	12619.4
Average	194604.9	205125.7	240867.9	289764.2	378403.3	565472.2	858459.6	1460604.1
P50	183004.6	192299.8	225788.1	271819.4	354544.8	528939.8	805844.7	1373089.6
P90	292188.1	308934.3	361530.2	435060.4	569214.8	848320.2	1288795.1	2196319.0
P95	351626.5	371746.5	432422.3	524562.4	684345.1	1030600.4	1539162.7	2604095.7
P99	555123.0	604472.1	719448.1	858295.7	1112878.5	1643599.7	2476371.9	4150556.0
P999	681548.8	716491.2	842323.4	1011852.9	1321871.6	1972664.0	3007519.3	5166714.8
Maximum	705785.9	742232.3	872021.2	1026426.2	1342664.0	1995493.6	3158377.3	5468721.5