LayerScale Benchmarks

Benchmarks

Measured, on the record

Every figure on this page is read from a benchmark result file measured on the same two NVIDIA H100s², LayerScale against vLLM and SGLang with the identical checkpoint on every engine. Where LayerScale is not the fastest, the page says so. The stateful half comes first: a live stream held in a session and queried while it grows. Then the conventional comparison, every model, every workload, with any failed measurement marked as excluded rather than charted.

0.383ms

Flash Query on a live 8k-token stream, p50¹

2.818ms/token

Single-stream decode, 128 in / 128 out, p50¹

90,229.87tok/s

Prefill, 4,096-token prompt¹

1.43x

faster agentic loop than SGLang, 5 turns on a 2,048-token prefix, 724.413 ms vs 1,034.626 ms (vLLM 888.67 ms)¹

Stateful

A live stream, queried while it grows

A stream is pushed into a LayerScale session 256 tokens at a time, every 250 ms, across an 8,192-token window, and a query is asked on an open loop at the same cadence. LayerScale answers two ways. A Flash Query is registered up front and answered from the standing session as the stream lands. A cold query arrives unannounced and is answered on the session that is already holding the stream. vLLM and SGLang have no session API, so serving this workload means re-sending the whole accumulated stream on every query. That is what the workload costs them, not a handicap imposed on them.

Flash Query on a live stream

end to end per query, ms, p50, lower is better. A 256-token entry lands every 250 ms across an 8,192-token window and a 16-token query is asked every 250 ms. The query is registered up front and answered from the standing session.

LayerScale, Flash Query answered from the standing sessionvLLM, stateless replaySGLang, stateless replay

Qwen3-0.6B: Flash 0.394 ms, vLLM 48.233 ms, SGLang 37.573 ms; Llama-3.2-3B-Instruct: Flash 0.415 ms, vLLM 60.886 ms, SGLang 55.077 ms; Qwen3-8B-FP8: Flash 0.383 ms, vLLM 86.215 ms, SGLang 72.468 ms; Qwen3-30B-A3B: Flash 0.387 ms, vLLM 124.199 ms, SGLang 88.909 ms.

Cold query on the same stream

end to end per query, ms, p50, lower is better. The query arrives unannounced and is answered on the session that is already holding the stream; vLLM and SGLang re-send the accumulated stream.

LayerScale, cold query on the standing sessionvLLM, stateless replaySGLang, stateless replay

Qwen3-0.6B: LayerScale 21.48, vLLM 48.233, SGLang 37.573 ms. Llama-3.2-3B-Instruct: LayerScale 34.334, vLLM 60.886, SGLang 55.077 ms. Qwen3-8B-FP8: LayerScale 53.562, vLLM 86.215, SGLang 72.468 ms. Qwen3-30B-A3B: LayerScale 58.621, vLLM 124.199, SGLang 88.909 ms. Qwen3-32B: LayerScale 248.178, vLLM 261.597, SGLang 263.295 ms.

0.394ms

Flash Query, Qwen3-0.6B

20 of 20 queries answered from the standing session

vLLM 48.233 ms · SGLang 37.573 ms, stateless replay

0.415ms

Flash Query, Llama-3.2-3B-Instruct

20 of 20 queries answered from the standing session

vLLM 60.886 ms · SGLang 55.077 ms, stateless replay

0.383ms

Flash Query, Qwen3-8B-FP8

20 of 20 queries answered from the standing session

vLLM 86.215 ms · SGLang 72.468 ms, stateless replay

0.387ms

Flash Query, Qwen3-30B-A3B

20 of 20 queries answered from the standing session

vLLM 124.199 ms · SGLang 88.909 ms, stateless replay

Qwen3-32B: no query in the Flash Query measurement was answered from the standing session (flash hits 0 of 20, fast_rate 0), so it has no Flash Query figure and is not in the first chart. It is in the cold comparison above and in the table, where its lead is narrower than the models that hit.

Two H100s (TP2) · Live stream, session-stream256-token entries every 250 ms, 8,192-token window, 16-token answers, 20 queries after 3 warm-up. n/a means that engine has no document for the row.
Workload	Metric	LayerScale	vLLM	SGLang
Qwen3-0.6B	Qwen3-0.6BFlash Query, answered from the standing session, ms flash hits 20 of 20, fast_rate 1	0.394 p95 0.572 · p99 0.585	n/a	n/a
Qwen3-0.6B	Qwen3-0.6BCold query, ms LayerScale on the standing session · vLLM and SGLang re-send the accumulated stream	21.48 p95 24.315 · p99 24.506	48.233 p95 51.372 · p99 51.841	37.573 p95 44.764 · p99 46.877
Qwen3-0.6B	Qwen3-0.6BTime to first token, stateless replay, ms LayerScale session answers are not streamed, so there is no first-token time to report	n/a	24.101 p95 26.462 · p99 26.938	15.972 p95 20.992 · p99 23.375
Llama-3.2-3B-Instruct	Llama-3.2-3B-InstructFlash Query, answered from the standing session, ms flash hits 20 of 20, fast_rate 1	0.415 p95 0.575 · p99 0.586	n/a	n/a
Llama-3.2-3B-Instruct	Llama-3.2-3B-InstructCold query, ms LayerScale on the standing session · vLLM and SGLang re-send the accumulated stream	34.334 p95 39.137 · p99 39.619	60.886 p95 64.627 · p99 65.545	55.077 p95 59.79 · p99 60.502
Llama-3.2-3B-Instruct	Llama-3.2-3B-InstructTime to first token, stateless replay, ms LayerScale session answers are not streamed, so there is no first-token time to report	n/a	23.332 p95 26.595 · p99 27.821	18.65 p95 22.633 · p99 22.961
Qwen3-8B-FP8	Qwen3-8B-FP8Flash Query, answered from the standing session, ms flash hits 20 of 20, fast_rate 1	0.383 p95 0.444 · p99 0.464	n/a	n/a
Qwen3-8B-FP8	Qwen3-8B-FP8Cold query, ms LayerScale on the standing session · vLLM and SGLang re-send the accumulated stream	53.562 p95 57.416 · p99 57.518	86.215 p95 93.003 · p99 94.052	72.468 p95 78.053 · p99 78.536
Qwen3-8B-FP8	Qwen3-8B-FP8Time to first token, stateless replay, ms LayerScale session answers are not streamed, so there is no first-token time to report	n/a	33.895 p95 38.999 · p99 39.762	20.906 p95 24.269 · p99 24.759
Qwen3-30B-A3B	Qwen3-30B-A3BFlash Query, answered from the standing session, ms flash hits 20 of 20, fast_rate 1	0.387 p95 0.569 · p99 0.586	n/a	n/a
Qwen3-30B-A3B	Qwen3-30B-A3BCold query, ms LayerScale on the standing session · vLLM and SGLang re-send the accumulated stream	58.621 p95 62.953 · p99 106.608	124.199 p95 509.116 · p99 520.222	88.909 p95 95.317 · p99 95.479
Qwen3-30B-A3B	Qwen3-30B-A3BTime to first token, stateless replay, ms LayerScale session answers are not streamed, so there is no first-token time to report	n/a	63.785 p95 449.29 · p99 460.37	26.595 p95 31.84 · p99 32.039
Qwen3-32B	Qwen3-32BFlash Query, answered from the standing session, ms flash hits 0 of 20, fast_rate 0 · no query was answered from the standing session in this measurement	n/a	n/a	n/a
Qwen3-32B	Qwen3-32BFlash Query measurement, every query built cold, ms the measurement fell through to the cold path throughout	222.384 p95 705.738 · p99 714.613	n/a	n/a
Qwen3-32B	Qwen3-32BCold query, ms LayerScale on the standing session · vLLM and SGLang re-send the accumulated stream	248.178 p95 695.986 · p99 708.497	261.597 p95 272.62 · p99 283.778	263.295 p95 270.336 · p99 279.796
Qwen3-32B	Qwen3-32BTime to first token, stateless replay, ms LayerScale session answers are not streamed, so there is no first-token time to report	n/a	48.924 p95 57.545 · p99 72.167	53.145 p95 56.471 · p99 72.755

Sessions for agents

The agentic loop held in a session

The same multi-turn loop, 48 output tokens per turn, driven through LayerScale's stateful session API instead of re-sending the conversation every turn. vLLM and SGLang have no equivalent endpoint, so the only like-for-like comparison is LayerScale against itself: the session loop beside the re-sent loop at the same turn count and prefix, both measured in this run. The re-sent loop ran with prefix caching on, so it already reuses the conversation prefix. At these turn counts the two loops run neck and neck; where they differ, the tables show by exactly how much. The turn-by-turn charts put the same loop on vLLM and SGLang beside them, so the first-token cost of an accumulating conversation is visible on every engine. With the prefix already cached, the session buys little on whole-loop time here, and at the last turn of the longest loop its first-token latency is higher than the re-sent loop on 4 of 5 models and lower on 1; the charts show it as measured.

Qwen3-0.6B

whole loop wall time, ms, p50, lower is better, turns · prefix

LayerScale, loop held in a sessionLayerScale, conversation re-sent each turn

10 turns · 8,192-token prefix: in a session 719.585 ms, re-sent 731.222 ms.

Qwen3-0.6B, turn by turn

time to first token per turn, ms, p50, 10 turns · 8,192-token prefix, lower is better

LayerScale, in a sessionLayerScale, re-sentvLLM, re-sentSGLang, re-sent

Turn 10: LayerScale, in a session 2.815 ms, LayerScale, re-sent 4.343 ms, vLLM, re-sent 29.214 ms, SGLang, re-sent 39.52 ms.

Llama-3.2-3B-Instruct

whole loop wall time, ms, p50, lower is better, turns · prefix

LayerScale, loop held in a sessionLayerScale, conversation re-sent each turn

10 turns · 8,192-token prefix: in a session 1,200.203 ms, re-sent 1,198.991 ms.

Llama-3.2-3B-Instruct, turn by turn

time to first token per turn, ms, p50, 10 turns · 8,192-token prefix, lower is better

LayerScale, in a sessionLayerScale, re-sentvLLM, re-sentSGLang, re-sent

Turn 10: LayerScale, in a session 6.219 ms, LayerScale, re-sent 5.539 ms, vLLM, re-sent 27.759 ms, SGLang, re-sent 41.947 ms.

Qwen3-8B-FP8

whole loop wall time, ms, p50, lower is better, turns · prefix

LayerScale, loop held in a sessionLayerScale, conversation re-sent each turn

10 turns · 8,192-token prefix: in a session 1,711.025 ms, re-sent 1,706.753 ms.

Qwen3-8B-FP8, turn by turn

time to first token per turn, ms, p50, 10 turns · 8,192-token prefix, lower is better

LayerScale, in a sessionLayerScale, re-sentvLLM, re-sentSGLang, re-sent

Turn 10: LayerScale, in a session 6.812 ms, LayerScale, re-sent 6.426 ms, vLLM, re-sent 30.832 ms, SGLang, re-sent 56.844 ms.

Qwen3-30B-A3B

whole loop wall time, ms, p50, lower is better, turns · prefix

LayerScale, loop held in a sessionLayerScale, conversation re-sent each turn

10 turns · 8,192-token prefix: in a session 1,801.02 ms, re-sent 1,768.197 ms.

Qwen3-30B-A3B, turn by turn

time to first token per turn, ms, p50, 10 turns · 8,192-token prefix, lower is better

LayerScale, in a sessionLayerScale, re-sentvLLM, re-sentSGLang, re-sent

Turn 10: LayerScale, in a session 14.156 ms, LayerScale, re-sent 10.878 ms, vLLM, re-sent 33.241 ms, SGLang, re-sent 78.826 ms.

Qwen3-32B

whole loop wall time, ms, p50, lower is better, turns · prefix

LayerScale, loop held in a sessionLayerScale, conversation re-sent each turn

10 turns · 8,192-token prefix: in a session 6,515.569 ms, re-sent 6,476.736 ms.

Qwen3-32B, turn by turn

time to first token per turn, ms, p50, 10 turns · 8,192-token prefix, lower is better

LayerScale, in a sessionLayerScale, re-sentvLLM, re-sentSGLang, re-sent

Turn 10: LayerScale, in a session 22.961 ms, LayerScale, re-sent 18.319 ms, vLLM, re-sent 42.074 ms, SGLang, re-sent 59.036 ms.

Two H100s (TP2) · Qwen3-0.6B · agentic loop in a session versus re-sentboth LayerScale, same build, same run, 48 output tokens per turn, 20 iterations after 3 warm-up
Workload	Metric	In a session	Re-sent each turn
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	51.463 p95 51.69 · p99 51.707	51.508 p95 51.859 · p99 51.915
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	2.151 p95 2.305 · p99 2.39	2.182 p95 2.49 · p99 2.537
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	51.312 p95 51.502 · p99 51.567	51.374 p95 51.686 · p99 51.769
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	71.873 p95 72.127 · p99 72.201	71.867 p95 71.95 · p99 71.98
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	3.942 p95 4.045 · p99 4.288	3.922 p95 3.972 · p99 4.01
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	71.647 p95 71.897 · p99 71.96	71.655 p95 71.704 · p99 71.763
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	262.788 p95 263.132 · p99 263.191	262.608 p95 262.996 · p99 263.066
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	2.268 p95 2.408 · p99 2.523	2.231 p95 2.529 · p99 2.65
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	52.357 p95 53.401 · p99 53.459	51.979 p95 53.602 · p99 53.797
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	356.126 p95 357.074 · p99 357.555	361.452 p95 361.826 · p99 361.933
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	2.838 p95 4.14 · p99 4.195	4.108 p95 4.241 · p99 4.356
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	71.084 p95 71.755 · p99 71.866	72.049 p95 72.614 · p99 72.717
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	538.969 p95 539.989 · p99 540.009	538.773 p95 539.334 · p99 539.433
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	2.29 p95 2.417 · p99 2.443	2.423 p95 2.59 · p99 2.639
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	53.555 p95 55.874 · p99 55.952	53.641 p95 55.992 · p99 56.095
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	719.585 p95 724.751 · p99 726.26	731.222 p95 732.136 · p99 732.567
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	2.846 p95 4.132 · p99 4.898	4.233 p95 4.373 · p99 4.436
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	71.714 p95 73.134 · p99 73.298	72.784 p95 74.579 · p99 74.655

Two H100s (TP2) · Llama-3.2-3B-Instruct · agentic loop in a session versus re-sentboth LayerScale, same build, same run, 48 output tokens per turn, 20 iterations after 3 warm-up
Workload	Metric	In a session	Re-sent each turn
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	98.695 p95 98.834 · p99 98.902	98.624 p95 98.896 · p99 98.956
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	3.336 p95 3.475 · p99 3.572	3.293 p95 3.353 · p99 3.588
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	98.522 p95 98.657 · p99 98.744	98.461 p95 98.707 · p99 98.783
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	118.909 p95 119.017 · p99 119.095	118.786 p95 118.861 · p99 118.884
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	5.145 p95 5.247 · p99 5.27	5.118 p95 5.17 · p99 5.189
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	118.685 p95 118.792 · p99 118.83	118.567 p95 118.635 · p99 118.663
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	500.551 p95 500.792 · p99 500.877	499.092 p95 499.62 · p99 499.894
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	3.78 p95 3.956 · p99 3.982	3.342 p95 3.609 · p99 3.707
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	99.851 p95 101.118 · p99 101.139	99.528 p95 100.892 · p99 100.961
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	594.362 p95 594.748 · p99 594.832	596.272 p95 596.951 · p99 597.24
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	4.385 p95 6.029 · p99 6.097	5.251 p95 5.352 · p99 5.442
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	118.515 p95 120.145 · p99 120.233	119.022 p95 119.489 · p99 119.598
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,016.056 p95 1,017.122 · p99 1,017.379	1,012.223 p95 1,013.43 · p99 1,013.827
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	3.884 p95 4.234 · p99 4.261	3.579 p95 3.719 · p99 3.773
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	101.198 p95 103.867 · p99 104.013	100.973 p95 103.321 · p99 103.415
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,200.203 p95 1,200.716 · p99 1,201.618	1,198.991 p95 1,199.831 · p99 1,199.845
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	6.01 p95 6.239 · p99 6.289	5.348 p95 5.54 · p99 5.585
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	120.062 p95 121.777 · p99 121.877	119.436 p95 121.234 · p99 121.318

Two H100s (TP2) · Qwen3-8B-FP8 · agentic loop in a session versus re-sentboth LayerScale, same build, same run, 48 output tokens per turn, 20 iterations after 3 warm-up
Workload	Metric	In a session	Re-sent each turn
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	143.556 p95 143.754 · p99 143.817	143.54 p95 143.606 · p99 144.126
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	4.174 p95 4.284 · p99 4.29	4.159 p95 4.206 · p99 4.211
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	143.344 p95 143.525 · p99 143.58	143.394 p95 143.464 · p99 143.99
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	169.361 p95 169.458 · p99 169.635	169.403 p95 169.521 · p99 169.61
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	5.879 p95 6.133 · p99 6.155	5.999 p95 6.093 · p99 6.098
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	169.128 p95 169.243 · p99 169.393	169.173 p95 169.302 · p99 169.367
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	729.939 p95 730.731 · p99 730.816	724.413 p95 724.89 · p99 725.052
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	5.776 p95 5.903 · p99 5.951	4.234 p95 4.507 · p99 4.555
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	145.823 p95 147.619 · p99 147.693	144.136 p95 146.371 · p99 146.462
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	851 p95 851.546 · p99 851.591	848.354 p95 848.783 · p99 848.889
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	6.834 p95 6.937 · p99 6.962	6.183 p95 6.328 · p99 6.411
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	169.957 p95 170.609 · p99 170.647	169.346 p95 170.084 · p99 170.13
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,479.254 p95 1,479.992 · p99 1,483.302	1,466.007 p95 1,466.463 · p99 1,466.828
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	5.848 p95 6.028 · p99 6.066	4.46 p95 4.65 · p99 4.745
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	147.765 p95 150.932 · p99 151.018	146.356 p95 149.562 · p99 149.667
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,711.025 p95 1,711.737 · p99 1,711.818	1,706.753 p95 1,707.462 · p99 1,707.682
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	6.839 p95 6.987 · p99 7.034	6.305 p95 6.493 · p99 6.605
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	170.678 p95 172.705 · p99 172.807	170.294 p95 172.374 · p99 172.482

Two H100s (TP2) · Qwen3-30B-A3B · agentic loop in a session versus re-sentboth LayerScale, same build, same run, 48 output tokens per turn, 20 iterations after 3 warm-up
Workload	Metric	In a session	Re-sent each turn
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	146.892 p95 147.316 · p99 147.351	146.788 p95 146.897 · p99 146.933
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	8.609 p95 8.771 · p99 8.776	8.463 p95 8.545 · p99 8.561
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	146.751 p95 147.122 · p99 147.182	146.639 p95 146.762 · p99 146.764
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	176.005 p95 176.35 · p99 176.823	176.088 p95 176.313 · p99 176.329
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	10.251 p95 10.579 · p99 10.649	10.255 p95 10.515 · p99 10.565
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	175.807 p95 176.115 · p99 176.6	175.855 p95 176.06 · p99 176.086
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	762.739 p95 763.653 · p99 763.911	735.053 p95 735.569 · p99 735.651
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	13.114 p95 13.355 · p99 13.399	5.985 p95 8.701 · p99 8.741
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	152.954 p95 155.073 · p99 155.147	147.217 p95 148.195 · p99 148.229
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	894.322 p95 896.008 · p99 896.053	879.651 p95 881.647 · p99 882.14
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	14.139 p95 14.54 · p99 14.641	10.361 p95 10.858 · p99 10.969
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	178.988 p95 180.188 · p99 180.329	175.665 p95 176.47 · p99 176.71
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,550.495 p95 1,552.376 · p99 1,552.608	1,489.762 p95 1,490.997 · p99 1,491.591
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	13.149 p95 13.358 · p99 13.399	6.052 p95 8.799 · p99 8.84
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	155.243 p95 158.754 · p99 158.906	148.538 p95 152.131 · p99 152.217
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,801.02 p95 1,805.585 · p99 1,805.973	1,768.197 p95 1,771.521 · p99 1,772.125
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	14.259 p95 14.628 · p99 14.706	10.706 p95 11.152 · p99 11.29
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	179.816 p95 182.351 · p99 182.597	176.375 p95 178.863 · p99 179.188

Two H100s (TP2) · Qwen3-32B · agentic loop in a session versus re-sentboth LayerScale, same build, same run, 48 output tokens per turn, 20 iterations after 3 warm-up
Workload	Metric	In a session	Re-sent each turn
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	620.877 p95 621.349 · p99 621.628	621.088 p95 621.646 · p99 621.65
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	15.619 p95 15.889 · p99 15.946	15.789 p95 15.947 · p99 16.009
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	620.666 p95 621.113 · p99 621.368	620.836 p95 621.382 · p99 621.387
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	644.865 p95 645.758 · p99 646.384	645.015 p95 645.495 · p99 645.564
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	17.95 p95 18.228 · p99 18.288	17.791 p95 18.177 · p99 18.25
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	644.528 p95 645.393 · p99 645.989	644.595 p95 645.093 · p99 645.118
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	3,131.735 p95 3,134.413 · p99 3,134.896	3,114.601 p95 3,115.934 · p99 3,116.048
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	19.728 p95 20.131 · p99 20.201	15.727 p95 16.109 · p99 16.138
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	626.414 p95 629.675 · p99 629.879	621.819 p95 625.215 · p99 625.41
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	3,249.284 p95 3,251.323 · p99 3,251.915	3,233.003 p95 3,235.299 · p99 3,235.686
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	22.426 p95 22.735 · p99 22.782	18.05 p95 18.429 · p99 18.478
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	650.299 p95 651.778 · p99 651.83	646.194 p95 647.575 · p99 647.656
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	6,300.715 p95 6,305.135 · p99 6,305.308	6,259.604 p95 6,263.516 · p99 6,264.447
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	19.942 p95 20.36 · p99 20.5	15.729 p95 16.036 · p99 16.146
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	629.878 p95 636.82 · p99 636.964	625.177 p95 632.261 · p99 632.541
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	6,515.569 p95 6,518.09 · p99 6,518.215	6,476.736 p95 6,479.159 · p99 6,479.461
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	22.684 p95 23.032 · p99 23.117	18.209 p95 18.556 · p99 18.689
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	651.584 p95 653.767 · p99 654.118	647.328 p95 649.292 · p99 649.559

Stateless

The conventional comparison

LayerScale, vLLM and SGLang on identical checkpoints, one model per section. Five workloads: a single stream, one request at a time; concurrent streams, each sending 512 prompt tokens and generating 128; prefill, a long prompt and 1 output token; an agentic loop that re-sends the accumulated conversation each turn and generates 48 tokens per turn; and the same loop with tool definitions attached and a 160-token budget per turn. Prefix caching is off on every engine for the single, concurrent and prefill workloads, which send one calibrated prompt repeatedly, and on for the two loops, where the growing conversation prefix is the workload.

Each section has the charts first and the full table beneath, every measured point, all three engines side by side, p50 in large type with p95 and p99 underneath. The best value in each row is marked, whichever engine produced it.

Model

Qwen3-0.6B

Qwen/Qwen3-0.6BTwo H100s (TP2)

LayerScale, vLLM and SGLang serving the identical checkpoint on the same two GPUs. Lower is better for every latency chart, higher is better for throughput. Where LayerScale is not the fastest, the chart and the table say so.

Decode latency under concurrency

ms per output token per request, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 4.082, vLLM 4.595, SGLang 4.283 ms/token. Every stream sends 512 prompt tokens and generates 128.

Time to first token under concurrency

ms, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 14.961, vLLM 235.522, SGLang 180.019 ms. Every stream sends 512 prompt tokens and generates 128.

Output throughput under concurrency

generated tokens per second over the window, higher is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 30,487.57, vLLM 19,895.59, SGLang 21,361.33 tok/s. Every stream sends 512 prompt tokens and generates 128.

Prefill throughput

prompt tokens per second, 1 output token, higher is better

LayerScalevLLMSGLang

On the 4,096-token prompt: LayerScale 341,412.16, vLLM 124,314.95, SGLang 156,922.62 tokens/s. On the 16,384-token prompt: LayerScale 266,833.48, vLLM 132,350.47, SGLang 155,847.01 tokens/s.

Single-stream latency

end to end per request, ms, p50, lower is better, prompt tokens → generated tokens

LayerScalevLLMSGLang

128 in / 128 out: LayerScale 121.334, vLLM 193.488, SGLang 184.702 ms. 4,096 in / 1,024 out: LayerScale 1,241.257, vLLM 1,647.419, SGLang 1,569.809 ms. Time to first token and per-token decode for every point are in the table.

Multi-turn agentic loop

whole loop wall time, ms, p50, 48 output tokens per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 731.222, vLLM 1,060.067, SGLang 1,134.593 ms.

Multi-turn tool loop

whole loop wall time with tool definitions attached, ms, p50, 160-token budget per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 2,389.952, vLLM 3,035.571, SGLang 3,018.876 ms.

Two H100s (TP2) · Single streamone request at a time, 20 iterations after 3 warm-up, p50 with p95 and p99 beneath
Workload	Metric	LayerScale	vLLM	SGLang
128 in / 128 out	128 in / 128 outTime to first token, ms	1.66 p95 1.893 · p99 1.903	7.516 p95 8.898 · p99 9.457	8.433 p95 9.702 · p99 10.133
128 in / 128 out	128 in / 128 outDecode, ms per token	0.942 p95 0.943 · p99 0.945	1.46 p95 1.466 · p99 1.47	1.386 p95 1.388 · p99 1.391
128 in / 128 out	128 in / 128 outEnd to end, ms	121.334 p95 121.646 · p99 121.874	193.488 p95 194.636 · p99 194.913	184.702 p95 185.775 · p99 185.952
128 in / 128 out	128 in / 128 outOutput tokens/s	1,053.53	661.3	691.84
128 in / 1,024 out	128 in / 1,024 outTime to first token, ms	1.862 p95 1.946 · p99 2.651	8.43 p95 8.726 · p99 9.104	9.316 p95 9.893 · p99 9.909
128 in / 1,024 out	128 in / 1,024 outDecode, ms per token	0.965 p95 0.966 · p99 0.966	1.487 p95 1.489 · p99 1.515	1.407 p95 1.408 · p99 1.408
128 in / 1,024 out	128 in / 1,024 outEnd to end, ms	988.796 p95 989.836 · p99 990.435	1,530.017 p95 1,531.778 · p99 1,557.931	1,448.599 p95 1,449.211 · p99 1,449.375
128 in / 1,024 out	128 in / 1,024 outOutput tokens/s	1,035.27	668.59	706.76
1,024 in / 128 out	1,024 in / 128 outTime to first token, ms	3.732 p95 3.8 · p99 4.024	16 p95 16.473 · p99 16.861	11.642 p95 12.255 · p99 12.437
1,024 in / 128 out	1,024 in / 128 outDecode, ms per token	0.991 p95 0.992 · p99 0.992	1.503 p95 1.506 · p99 1.507	1.413 p95 1.415 · p99 1.419
1,024 in / 128 out	1,024 in / 128 outEnd to end, ms	129.516 p95 129.692 · p99 129.96	206.866 p95 207.522 · p99 207.75	191.301 p95 191.64 · p99 191.899
1,024 in / 128 out	1,024 in / 128 outOutput tokens/s	987.31	618.06	669.47
1,024 in / 1,024 out	1,024 in / 1,024 outTime to first token, ms	3.799 p95 3.903 · p99 3.932	17.793 p95 18.295 · p99 18.328	11.803 p95 13.049 · p99 13.449
1,024 in / 1,024 out	1,024 in / 1,024 outDecode, ms per token	1.015 p95 1.016 · p99 1.016	1.514 p95 1.515 · p99 1.515	1.434 p95 1.444 · p99 1.472
1,024 in / 1,024 out	1,024 in / 1,024 outEnd to end, ms	1,042.419 p95 1,043.381 · p99 1,043.413	1,566.896 p95 1,568.179 · p99 1,568.363	1,478.739 p95 1,490.556 · p99 1,518.119
1,024 in / 1,024 out	1,024 in / 1,024 outOutput tokens/s	982.05	653.45	691.08
4,096 in / 128 out	4,096 in / 128 outTime to first token, ms	11.456 p95 11.728 · p99 11.739	35.845 p95 36.574 · p99 36.745	23.539 p95 24.455 · p99 24.883
4,096 in / 128 out	4,096 in / 128 outDecode, ms per token	1.166 p95 1.167 · p99 1.167	1.559 p95 1.561 · p99 1.561	1.493 p95 1.495 · p99 1.507
4,096 in / 128 out	4,096 in / 128 outEnd to end, ms	159.588 p95 159.772 · p99 159.778	233.674 p95 234.509 · p99 234.61	213.212 p95 214.478 · p99 215.403
4,096 in / 128 out	4,096 in / 128 outOutput tokens/s	801.25	549.05	599.32
4,096 in / 1,024 out	4,096 in / 1,024 outTime to first token, ms	12.027 p95 12.227 · p99 12.259	35.632 p95 37.686 · p99 38.311	26.2 p95 27.062 · p99 27.668
4,096 in / 1,024 out	4,096 in / 1,024 outDecode, ms per token	1.202 p95 1.202 · p99 1.202	1.575 p95 1.577 · p99 1.577	1.509 p95 1.509 · p99 1.57
4,096 in / 1,024 out	4,096 in / 1,024 outEnd to end, ms	1,241.257 p95 1,242.166 · p99 1,242.278	1,647.419 p95 1,650.006 · p99 1,650.803	1,569.809 p95 1,571.294 · p99 1,632.474
4,096 in / 1,024 out	4,096 in / 1,024 outOutput tokens/s	824.8	621.41	651.03

Two H100s (TP2) · Concurrent streamsevery stream sends 512 prompt tokens and generates 128, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
1 stream	1 streamTime to first token, ms	2.497 p95 2.53 · p99 2.573	14.875 p95 15.586 · p99 17.533	9.554 p95 10.722 · p99 11.26
1 stream	1 streamDecode, ms per token	0.96 p95 0.961 · p99 0.961	1.483 p95 1.489 · p99 1.49	1.4 p95 1.402 · p99 1.404
1 stream	1 streamEnd to end, ms	124.472 p95 124.545 · p99 124.615	203.226 p95 204.568 · p99 206.571	187.414 p95 188.346 · p99 188.555
1 stream	1 streamOutput tokens/s	1,028.12	628.74	681.94
1 stream	1 streamRequests/s	8.032	4.912	5.328
8 streams	8 streamsTime to first token, ms	8.097 p95 9.043 · p99 10.297	31.388 p95 38.038 · p99 39.929	21.316 p95 26.938 · p99 319.363
8 streams	8 streamsDecode, ms per token	1.261 p95 1.291 · p99 1.292	1.632 p95 1.764 · p99 1.768	1.609 p95 1.654 · p99 1.695
8 streams	8 streamsEnd to end, ms	166.83 p95 168.211 · p99 168.31	241.218 p95 245.734 · p99 246.299	225.058 p95 230.557 · p99 359.183
8 streams	8 streamsOutput tokens/s	6,117.66	4,230.45	4,391.44
8 streams	8 streamsRequests/s	47.794	33.05	34.308
32 streams	32 streamsTime to first token, ms	10.606 p95 17.791 · p99 40.206	79.085 p95 107.737 · p99 115.526	56.742 p95 63.68 · p99 64.94
32 streams	32 streamsDecode, ms per token	1.892 p95 1.922 · p99 1.938	2.368 p95 2.606 · p99 2.679	2.076 p95 2.094 · p99 2.101
32 streams	32 streamsEnd to end, ms	250.867 p95 258.17 · p99 283.739	373.521 p95 385.95 · p99 389.437	319.943 p95 323.246 · p99 327.162
32 streams	32 streamsOutput tokens/s	16,237.56	10,890.43	12,745.38
32 streams	32 streamsRequests/s	126.856	85.082	99.573
64 streams	64 streamsTime to first token, ms	12.334 p95 19.72 · p99 75.944	130.803 p95 195.436 · p99 205.093	111.648 p95 384.229 · p99 475.393
64 streams	64 streamsDecode, ms per token	2.603 p95 2.652 · p99 2.667	3.142 p95 3.577 · p99 3.676	2.628 p95 3.031 · p99 5.058
64 streams	64 streamsEnd to end, ms	344.705 p95 351.811 · p99 405.445	528.091 p95 540.703 · p99 542.816	444.21 p95 700.954 · p99 706.545
64 streams	64 streamsOutput tokens/s	23,671.07	15,447.27	17,369.67
64 streams	64 streamsRequests/s	184.93	120.682	135.701
128 streams	128 streamsTime to first token, ms	14.961 p95 22.883 · p99 170.236	235.522 p95 372.961 · p99 388.958	180.019 p95 256.992 · p99 638.701
128 streams	128 streamsDecode, ms per token	4.082 p95 4.108 · p99 4.124	4.595 p95 5.414 · p99 5.515	4.283 p95 7.104 · p99 7.625
128 streams	128 streamsEnd to end, ms	533.859 p95 542.145 · p99 678.146	819.841 p95 840.387 · p99 848.143	698.045 p95 1,107.129 · p99 1,329.749
128 streams	128 streamsOutput tokens/s	30,487.57	19,895.59	21,361.33
128 streams	128 streamsRequests/s	238.184	155.434	166.885

Two H100s (TP2) · Prefillone prompt, 1 output token, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
4,096-token prompt	4,096-token promptPrompt tokens/s	341,412.16	124,314.95	156,922.62
4,096-token prompt	4,096-token promptTime to first token, ms	11.808 p95 11.851 · p99 11.856	32.662 p95 33.883 · p99 34.111	25.631 p95 27.619 · p99 29.32
16,384-token prompt	16,384-token promptPrompt tokens/s	266,833.48	132,350.47	155,847.01
16,384-token prompt	16,384-token promptTime to first token, ms	61.039 p95 61.326 · p99 61.367	122.816 p95 127.277 · p99 128.314	104.601 p95 106.529 · p99 108.042

Two H100s (TP2) · Agentic loopeach turn re-sends the accumulated conversation and generates 48 tokens
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	51.508 p95 51.859 · p99 51.915	82.851 p95 83.377 · p99 83.906	91.762 p95 92.3 · p99 92.82
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	2.182 p95 2.49 · p99 2.537	11.131 p95 11.74 · p99 12.18	24.706 p95 25.342 · p99 25.546
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	51.374 p95 51.686 · p99 51.769	82.639 p95 83.162 · p99 83.703	91.563 p95 92.102 · p99 92.615
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	71.867 p95 71.95 · p99 71.98	102.991 p95 107.378 · p99 107.626	110.064 p95 110.875 · p99 110.986
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	3.922 p95 3.972 · p99 4.01	25.771 p95 29.99 · p99 31.722	35.622 p95 36.822 · p99 36.997
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	71.655 p95 71.704 · p99 71.763	102.703 p95 107.017 · p99 107.24	109.778 p95 110.574 · p99 110.673
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	262.608 p95 262.996 · p99 263.066	419.209 p95 426.546 · p99 429.103	465.243 p95 467.419 · p99 467.478
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	2.231 p95 2.529 · p99 2.65	11.809 p95 13.733 · p99 14.989	25.184 p95 26.353 · p99 26.807
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	51.979 p95 53.602 · p99 53.797	83.501 p95 85.878 · p99 87.131	92.904 p95 94.366 · p99 94.66
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	361.452 p95 361.826 · p99 361.933	521.825 p95 534.766 · p99 536.66	559.469 p95 564.954 · p99 565.994
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	4.108 p95 4.241 · p99 4.356	26.732 p95 29.996 · p99 30.69	37.095 p95 38.963 · p99 40.168
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	72.049 p95 72.614 · p99 72.717	104.324 p95 107.441 · p99 108.332	111.598 p95 113.409 · p99 115.2
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	538.773 p95 539.334 · p99 539.433	858.119 p95 873.628 · p99 873.752	950.421 p95 970.478 · p99 970.871
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	2.423 p95 2.59 · p99 2.639	13.51 p95 15.982 · p99 16.433	26.594 p95 29.054 · p99 29.798
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	53.641 p95 55.992 · p99 56.095	86.042 p95 89.033 · p99 89.74	95.065 p95 98.455 · p99 99.297
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	731.222 p95 732.136 · p99 732.567	1,060.067 p95 1,089.866 · p99 1,090.652	1,134.593 p95 1,164.007 · p99 1,167.539
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	4.233 p95 4.373 · p99 4.436	27.858 p95 31.767 · p99 32.583	38.042 p95 42.015 · p99 43.083
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	72.784 p95 74.579 · p99 74.655	106.266 p95 110.041 · p99 111.391	113.542 p95 117.806 · p99 119.376

Two H100s (TP2) · Tool loopthe same loop with tool definitions attached and a 160-token budget per turn
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	169.168 p95 169.611 · p99 169.783	256.602 p95 258.031 · p99 258.255	257.582 p95 259.237 · p99 259.315
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	2.321 p95 2.524 · p99 2.603	13.874 p95 15.216 · p99 15.331	27.276 p95 29.083 · p99 29.518
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	169.014 p95 169.374 · p99 169.608	256.395 p95 257.818 · p99 258.024	257.371 p95 258.955 · p99 259.015
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	233.625 p95 233.698 · p99 233.706	293.713 p95 295.397 · p99 297.282	293.577 p95 297.503 · p99 297.789
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	3.992 p95 4.036 · p99 4.092	26.495 p95 28.722 · p99 30.06	39.802 p95 43.706 · p99 43.915
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	233.396 p95 233.461 · p99 233.464	293.409 p95 295.04 · p99 296.928	293.2 p95 297.083 · p99 297.36
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	872.921 p95 874.051 · p99 874.763	1,297.115 p95 1,305.487 · p99 1,306.317	1,308.416 p95 1,320.735 · p99 1,328.254
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	2.43 p95 2.657 · p99 2.707	13.832 p95 16.734 · p99 17.341	28.944 p95 32.101 · p99 33.629
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	174.031 p95 179.893 · p99 180.013	258.6 p95 265.018 · p99 265.728	261.473 p95 266.904 · p99 269.156
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	1,180.576 p95 1,182.116 · p99 1,182.514	1,493.762 p95 1,511.99 · p99 1,512.691	1,488.802 p95 1,502.965 · p99 1,505.063
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	4.254 p95 4.504 · p99 4.641	29.634 p95 33.368 · p99 34.643	40.936 p95 45.44 · p99 46.963
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	234.773 p95 241.622 · p99 241.888	298.99 p95 304.545 · p99 306.671	297.76 p95 303.317 · p99 305.163
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,808.28 p95 1,810.816 · p99 1,810.982	2,630.698 p95 2,649.858 · p99 2,650.832	2,655.446 p95 2,682.202 · p99 2,682.691
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	2.646 p95 2.906 · p99 3.014	15.615 p95 19.844 · p99 21.358	30.14 p95 34.101 · p99 35.135
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	180.216 p95 192.258 · p99 192.538	264.672 p95 270.09 · p99 272.351	266.644 p95 272.482 · p99 275.183
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	2,389.952 p95 2,392.675 · p99 2,392.932	3,035.571 p95 3,070.281 · p99 3,071.841	3,018.876 p95 3,043.409 · p99 3,055.218
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	4.434 p95 4.744 · p99 4.831	30.461 p95 35.255 · p99 36.919	41.397 p95 46.038 · p99 47.165
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	237.941 p95 243.738 · p99 243.995	303.488 p95 312.58 · p99 316.498	301.647 p95 310.84 · p99 313.167

Model

Llama-3.2-3B-Instruct

meta-llama/Llama-3.2-3B-InstructTwo H100s (TP2)

Decode latency under concurrency

ms per output token per request, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 6.725, vLLM 6.853, SGLang 6.67 ms/token. SGLang leads at 128 streams. Every stream sends 512 prompt tokens and generates 128.

Time to first token under concurrency

ms, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 29.964, vLLM 194.583, SGLang 190.875 ms. Every stream sends 512 prompt tokens and generates 128.

Output throughput under concurrency

generated tokens per second over the window, higher is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 18,372.42, vLLM 15,579.36, SGLang 15,562.78 tok/s. Every stream sends 512 prompt tokens and generates 128.

Prefill throughput

prompt tokens per second, 1 output token, higher is better

LayerScalevLLMSGLang

On the 4,096-token prompt: LayerScale 134,299.3, vLLM 94,537.16, SGLang 90,540.2 tokens/s. On the 16,384-token prompt: LayerScale 105,927.21, vLLM 81,074.51, SGLang 83,857.16 tokens/s.

Single-stream latency

end to end per request, ms, p50, lower is better, prompt tokens → generated tokens

LayerScalevLLMSGLang

128 in / 128 out: LayerScale 246.55, vLLM 328.317, SGLang 317.743 ms. 4,096 in / 1,024 out: LayerScale 2,267.57, vLLM 2,713.05, SGLang 2,627.26 ms. Time to first token and per-token decode for every point are in the table.

Multi-turn agentic loop

whole loop wall time, ms, p50, 48 output tokens per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 1,198.991, vLLM 1,506.477, SGLang 1,632.145 ms.

Multi-turn tool loop

whole loop wall time with tool definitions attached, ms, p50, 160-token budget per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 721.201, vLLM 979.747, SGLang 1,258.906 ms.

Two H100s (TP2) · Single streamone request at a time, 20 iterations after 3 warm-up, p50 with p95 and p99 beneath
Workload	Metric	LayerScale	vLLM	SGLang
128 in / 128 out	128 in / 128 outTime to first token, ms	2.964 p95 3.004 · p99 3.018	7.221 p95 8.531 · p99 8.982	9.43 p95 10.819 · p99 10.873
128 in / 128 out	128 in / 128 outDecode, ms per token	1.918 p95 1.919 · p99 1.92	2.528 p95 2.529 · p99 2.529	2.427 p95 2.431 · p99 2.431
128 in / 128 out	128 in / 128 outEnd to end, ms	246.55 p95 246.667 · p99 246.783	328.317 p95 329.511 · p99 330.123	317.743 p95 318.373 · p99 318.629
128 in / 128 out	128 in / 128 outOutput tokens/s	519	389.36	402.6
128 in / 1,024 out	128 in / 1,024 outTime to first token, ms	3.093 p95 3.152 · p99 3.182	8.194 p95 8.664 · p99 8.7	9.673 p95 10.915 · p99 11.02
128 in / 1,024 out	128 in / 1,024 outDecode, ms per token	1.942 p95 1.944 · p99 1.944	2.534 p95 2.536 · p99 2.537	2.446 p95 2.448 · p99 2.449
128 in / 1,024 out	128 in / 1,024 outEnd to end, ms	1,989.891 p95 1,991.23 · p99 1,991.861	2,600.417 p95 2,602.208 · p99 2,603.316	2,512.441 p95 2,514.412 · p99 2,514.426
128 in / 1,024 out	128 in / 1,024 outOutput tokens/s	514.53	393.76	407.57
1,024 in / 128 out	1,024 in / 128 outTime to first token, ms	7.824 p95 7.893 · p99 7.962	15.672 p95 17.311 · p99 18.14	13.65 p95 15.643 · p99 16.104
1,024 in / 128 out	1,024 in / 128 outDecode, ms per token	1.969 p95 1.97 · p99 1.97	2.539 p95 2.544 · p99 2.544	2.451 p95 2.455 · p99 2.457
1,024 in / 128 out	1,024 in / 128 outEnd to end, ms	257.899 p95 257.972 · p99 258.192	338.097 p95 339.752 · p99 340.149	324.829 p95 327.272 · p99 327.583
1,024 in / 128 out	1,024 in / 128 outOutput tokens/s	496.21	377.92	393.24
1,024 in / 1,024 out	1,024 in / 1,024 outTime to first token, ms	8.315 p95 8.602 · p99 8.936	16.657 p95 17.526 · p99 18.339	15.516 p95 15.976 · p99 16.355
1,024 in / 1,024 out	1,024 in / 1,024 outDecode, ms per token	1.994 p95 1.996 · p99 2	2.55 p95 2.552 · p99 2.552	2.463 p95 2.465 · p99 2.465
1,024 in / 1,024 out	1,024 in / 1,024 outEnd to end, ms	2,048.675 p95 2,050.027 · p99 2,054.7	2,625.43 p95 2,628.01 · p99 2,628.137	2,535.08 p95 2,536.223 · p99 2,536.381
1,024 in / 1,024 out	1,024 in / 1,024 outOutput tokens/s	499.79	389.97	403.91
4,096 in / 128 out	4,096 in / 128 outTime to first token, ms	29.571 p95 29.683 · p99 29.701	42.766 p95 44.068 · p99 44.164	42.972 p95 44.887 · p99 45.366
4,096 in / 128 out	4,096 in / 128 outDecode, ms per token	2.152 p95 2.156 · p99 2.157	2.593 p95 2.598 · p99 2.598	2.511 p95 2.514 · p99 2.515
4,096 in / 128 out	4,096 in / 128 outEnd to end, ms	302.912 p95 303.365 · p99 303.446	371.937 p95 373.205 · p99 374.016	361.996 p95 363.325 · p99 364.105
4,096 in / 128 out	4,096 in / 128 outOutput tokens/s	422.22	343.86	353.26
4,096 in / 1,024 out	4,096 in / 1,024 outTime to first token, ms	29.7 p95 29.897 · p99 30.046	44.447 p95 45.126 · p99 45.487	43.229 p95 45.132 · p99 46.058
4,096 in / 1,024 out	4,096 in / 1,024 outDecode, ms per token	2.188 p95 2.19 · p99 2.191	2.609 p95 2.612 · p99 2.612	2.526 p95 2.528 · p99 2.528
4,096 in / 1,024 out	4,096 in / 1,024 outEnd to end, ms	2,267.57 p95 2,270.016 · p99 2,271.156	2,713.05 p95 2,716.277 · p99 2,716.891	2,627.26 p95 2,629.646 · p99 2,630.889
4,096 in / 1,024 out	4,096 in / 1,024 outOutput tokens/s	451.49	377.37	389.72

Two H100s (TP2) · Concurrent streamsevery stream sends 512 prompt tokens and generates 128, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
1 stream	1 streamTime to first token, ms	4.858 p95 4.894 · p99 4.934	15.918 p95 17.008 · p99 17.079	9.403 p95 9.701 · p99 10.729
1 stream	1 streamDecode, ms per token	1.937 p95 1.938 · p99 1.938	2.532 p95 2.534 · p99 2.534	2.44 p95 2.443 · p99 2.443
1 stream	1 streamEnd to end, ms	250.82 p95 250.993 · p99 251.019	337.056 p95 338.593 · p99 338.618	319.306 p95 319.851 · p99 320.211
1 stream	1 streamOutput tokens/s	510.15	379.43	400.58
1 stream	1 streamRequests/s	3.986	2.964	3.13
8 streams	8 streamsTime to first token, ms	17.473 p95 25.865 · p99 26.843	35.338 p95 38.819 · p99 46.435	29.077 p95 36.056 · p99 36.961
8 streams	8 streamsDecode, ms per token	2.308 p95 2.379 · p99 2.404	2.696 p95 2.864 · p99 2.866	2.779 p95 2.881 · p99 2.886
8 streams	8 streamsEnd to end, ms	315.056 p95 316.227 · p99 316.269	379.878 p95 381.116 · p99 387.308	381.973 p95 383.209 · p99 383.424
8 streams	8 streamsOutput tokens/s	3,246.4	2,691.29	2,679.47
8 streams	8 streamsRequests/s	25.362	21.026	20.933
32 streams	32 streamsTime to first token, ms	21.902 p95 40.461 · p99 97.309	53.7 p95 74.781 · p99 131.603	72.291 p95 115.641 · p99 117.579
32 streams	32 streamsDecode, ms per token	3.227 p95 3.297 · p99 3.335	3.576 p95 3.77 · p99 3.858	3.451 p95 3.561 · p99 3.569
32 streams	32 streamsEnd to end, ms	431.731 p95 445.671 · p99 502.162	507.746 p95 524.491 · p99 573.543	509.873 p95 512.3 · p99 512.499
32 streams	32 streamsOutput tokens/s	9,434.8	8,022.56	8,021.5
32 streams	32 streamsRequests/s	73.709	62.676	62.668
64 streams	64 streamsTime to first token, ms	25.651 p95 42.209 · p99 180.784	77.226 p95 137.863 · p99 233.178	120.989 p95 167.329 · p99 441.066
64 streams	64 streamsDecode, ms per token	4.339 p95 4.407 · p99 4.46	4.754 p95 5.045 · p99 5.143	4.366 p95 4.562 · p99 6.71
64 streams	64 streamsEnd to end, ms	576.976 p95 592.376 · p99 727.013	679.823 p95 743.81 · p99 792.385	676.355 p95 729.509 · p99 987.707
64 streams	64 streamsOutput tokens/s	14,105.33	11,985.51	11,842.37
64 streams	64 streamsRequests/s	110.198	93.637	92.518
128 streams	128 streamsTime to first token, ms	29.964 p95 46.623 · p99 379.934	194.583 p95 416.593 · p99 473.472	190.875 p95 525.328 · p99 572.514
128 streams	128 streamsDecode, ms per token	6.725 p95 6.781 · p99 6.827 SGLang leads	6.853 p95 7.478 · p99 7.641	6.67 p95 7.923 · p99 9.635
128 streams	128 streamsEnd to end, ms	885.298 p95 902.802 · p99 1,227.29	1,056.168 p95 1,288.094 · p99 1,344.583	1,035.75 p95 1,208.494 · p99 1,389.775
128 streams	128 streamsOutput tokens/s	18,372.42	15,579.36	15,562.78
128 streams	128 streamsRequests/s	143.535	121.714	121.584

Two H100s (TP2) · Prefillone prompt, 1 output token, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
4,096-token prompt	4,096-token promptPrompt tokens/s	134,299.3	94,537.16	90,540.2
4,096-token prompt	4,096-token promptTime to first token, ms	30.314 p95 30.511 · p99 30.626	43.14 p95 43.443 · p99 43.542	44.854 p95 46.368 · p99 47.203
16,384-token prompt	16,384-token promptPrompt tokens/s	105,927.21	81,074.51	83,857.16
16,384-token prompt	16,384-token promptTime to first token, ms	154.399 p95 155.709 · p99 156.031	200.731 p95 206.637 · p99 209.588	194.877 p95 197.192 · p99 201.394

Two H100s (TP2) · Agentic loopeach turn re-sends the accumulated conversation and generates 48 tokens
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	98.624 p95 98.896 · p99 98.956	131.171 p95 132.037 · p99 132.921	142.613 p95 143.971 · p99 144.114
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	3.293 p95 3.353 · p99 3.588	11.015 p95 12.051 · p99 13.44	27.278 p95 28.926 · p99 28.988
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	98.461 p95 98.707 · p99 98.783	131.018 p95 131.881 · p99 132.75	142.38 p95 143.735 · p99 143.834
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	118.786 p95 118.861 · p99 118.884	148.537 p95 149.182 · p99 149.411	160.626 p95 163.934 · p99 166.94
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	5.118 p95 5.17 · p99 5.189	24.324 p95 25.208 · p99 25.377	39.231 p95 42.432 · p99 45.101
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	118.567 p95 118.635 · p99 118.663	148.261 p95 148.885 · p99 149.042	160.313 p95 163.619 · p99 166.578
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	499.092 p95 499.62 · p99 499.894	661.881 p95 668.923 · p99 670.251	707.271 p95 723.575 · p99 724.268
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	3.342 p95 3.609 · p99 3.707	12.289 p95 14.341 · p99 14.768	26.227 p95 30.277 · p99 30.52
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	99.528 p95 100.892 · p99 100.961	132.34 p95 134.476 · p99 134.741	141.611 p95 145.362 · p99 146.069
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	596.272 p95 596.951 · p99 597.24	748.262 p95 761.009 · p99 761.446	808.703 p95 846.385 · p99 849.209
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	5.251 p95 5.352 · p99 5.442	25.553 p95 29.049 · p99 29.68	40.096 p95 47.546 · p99 49.181
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	119.022 p95 119.489 · p99 119.598	149.808 p95 152.628 · p99 153.198	161.899 p95 169.603 · p99 171.399
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,012.223 p95 1,013.43 · p99 1,013.827	1,333.92 p95 1,350.302 · p99 1,350.78	1,428.755 p95 1,462.253 · p99 1,466.975
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	3.579 p95 3.719 · p99 3.773	13.374 p95 15.666 · p99 16.323	27.103 p95 31.579 · p99 32.903
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	100.973 p95 103.321 · p99 103.415	133.496 p95 136.403 · p99 136.949	143.275 p95 147.829 · p99 149.5
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,198.991 p95 1,199.831 · p99 1,199.845	1,506.477 p95 1,511.68 · p99 1,511.716	1,632.145 p95 1,682.541 · p99 1,707.104
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	5.348 p95 5.54 · p99 5.585	26.033 p95 27.948 · p99 28.196	40.756 p95 48.802 · p99 50.487
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	119.436 p95 121.234 · p99 121.318	150.399 p95 152.9 · p99 153.183	163.257 p95 170.678 · p99 173.416

Two H100s (TP2) · Tool loopthe same loop with tool definitions attached and a 160-token budget per turn
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	66.75 p95 66.939 · p99 67.849	90.729 p95 91.187 · p99 91.448	77.207 p95 77.898 · p99 79.228
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	66.493 p95 66.682 · p99 67.573 vLLM leads	26.311 p95 26.777 · p99 27.167	39.738 p95 40.443 · p99 41.607
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	66.493 p95 66.682 · p99 67.573	90.58 p95 91.001 · p99 91.253	76.975 p95 77.667 · p99 79.02
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	58.781 p95 58.956 · p99 59.026	81.802 p95 83.27 · p99 83.7	94.738 p95 100.8 · p99 104.102
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	58.437 p95 58.621 · p99 58.69 vLLM leads	40.857 p95 42.239 · p99 42.667	55.011 p95 60.786 · p99 64.13
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	58.437 p95 58.621 · p99 58.69	81.46 p95 82.913 · p99 83.318	94.413 p95 100.407 · p99 103.709
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	336.491 p95 336.76 · p99 336.776	457.291 p95 458.284 · p99 458.301	518.134 p95 524.78 · p99 533.173
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	66.575 p95 72.969 · p99 73.04 vLLM leads	27.048 p95 29.686 · p99 30.466	41.974 p95 44.454 · p99 45.94
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	70.892 p95 72.969 · p99 73.04	95.369 p95 98.929 · p99 99.781	109.359 p95 111.857 · p99 113.217
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	345.955 p95 346.369 · p99 346.39	471.062 p95 475.189 · p99 475.291	607.645 p95 614.19 · p99 615.009
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	58.513 p95 87.941 · p99 87.998 vLLM leads	39.663 p95 41.79 · p99 42.701	56.132 p95 58.638 · p99 60.241
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	58.513 p95 87.941 · p99 87.998	81.66 p95 114.727 · p99 115.398	127.495 p95 129.868 · p99 131.285
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	791.332 p95 791.737 · p99 791.737	1,071.965 p95 1,091.279 · p99 1,093.155	1,069.718 p95 1,079.772 · p99 1,080.025
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	53.848 p95 73.092 · p99 73.174 vLLM leads	27.769 p95 30.567 · p99 31.373	42.162 p95 43.876 · p99 44.597
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	71.041 p95 136.985 · p99 137.087	97.011 p95 182.497 · p99 183.872	109.829 p95 111.997 · p99 112.669
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	721.201 p95 721.789 · p99 722.033	979.747 p95 1,006.515 · p99 1,007.496	1,258.906 p95 1,327.217 · p99 1,332.718
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	58.718 p95 87.867 · p99 87.949 vLLM leads	40.8 p95 43.897 · p99 44.544	57.268 p95 64.007 · p99 66.804
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	58.718 p95 87.867 · p99 87.949	85.977 p95 116.967 · p99 118.296	128.621 p95 135.607 · p99 138.889

Model

Qwen3-8B-FP8

Qwen/Qwen3-8B-FP8Two H100s (TP2)

Decode latency under concurrency

ms per output token per request, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 10.057, vLLM 11.602, SGLang 10.277 ms/token. SGLang leads at 64 streams. Every stream sends 512 prompt tokens and generates 128.

Time to first token under concurrency

ms, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 44.528, vLLM 142.236, SGLang 219.642 ms. Every stream sends 512 prompt tokens and generates 128.

Output throughput under concurrency

generated tokens per second over the window, higher is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 12,307.86, vLLM 10,071.74, SGLang 10,682.72 tok/s. Every stream sends 512 prompt tokens and generates 128.

Prefill throughput

prompt tokens per second, 1 output token, higher is better

LayerScalevLLMSGLang

On the 4,096-token prompt: LayerScale 90,229.87, vLLM 60,141.9, SGLang 70,305.79 tokens/s. On the 16,384-token prompt: LayerScale 70,978.23, vLLM 52,916.46, SGLang 60,031.58 tokens/s.

Single-stream latency

end to end per request, ms, p50, lower is better, prompt tokens → generated tokens

LayerScalevLLMSGLang

128 in / 128 out: LayerScale 362.047, vLLM 444.143, SGLang 431.902 ms. 4,096 in / 1,024 out: LayerScale 3,288.787, vLLM 3,717.26, SGLang 3,630.023 ms. Time to first token and per-token decode for every point are in the table.

Multi-turn agentic loop

whole loop wall time, ms, p50, 48 output tokens per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 1,706.753, vLLM 2,008.225, SGLang 2,248.313 ms.

Multi-turn tool loop

whole loop wall time with tool definitions attached, ms, p50, 160-token budget per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 5,221.297, vLLM 5,785.214, SGLang 5,938.374 ms.

Two H100s (TP2) · Single streamone request at a time, 20 iterations after 3 warm-up, p50 with p95 and p99 beneath
Workload	Metric	LayerScale	vLLM	SGLang
128 in / 128 out	128 in / 128 outTime to first token, ms	4.261 p95 4.313 · p99 4.353	9.266 p95 9.88 · p99 10.355	11.454 p95 12.556 · p99 12.838
128 in / 128 out	128 in / 128 outDecode, ms per token	2.818 p95 2.819 · p99 2.819	3.423 p95 3.427 · p99 3.43	3.31 p95 3.312 · p99 3.313
128 in / 128 out	128 in / 128 outEnd to end, ms	362.047 p95 362.237 · p99 362.24	444.143 p95 445.086 · p99 446.024	431.902 p95 432.754 · p99 432.956
128 in / 128 out	128 in / 128 outOutput tokens/s	353.41	288.2	296.23
128 in / 1,024 out	128 in / 1,024 outTime to first token, ms	4.321 p95 4.403 · p99 4.405	9.419 p95 9.896 · p99 10.461	11.544 p95 12.106 · p99 12.271
128 in / 1,024 out	128 in / 1,024 outDecode, ms per token	2.852 p95 2.856 · p99 2.856	3.451 p95 3.453 · p99 3.454	3.348 p95 3.35 · p99 3.35
128 in / 1,024 out	128 in / 1,024 outEnd to end, ms	2,922.243 p95 2,925.916 · p99 2,926.348	3,539.745 p95 3,542.32 · p99 3,542.339	3,436.67 p95 3,438.602 · p99 3,438.94
128 in / 1,024 out	128 in / 1,024 outOutput tokens/s	350.36	289.27	297.94
1,024 in / 128 out	1,024 in / 128 outTime to first token, ms	11.693 p95 11.806 · p99 11.813	26.506 p95 26.979 · p99 28.913	24.198 p95 25.341 · p99 25.421
1,024 in / 128 out	1,024 in / 128 outDecode, ms per token	2.883 p95 2.886 · p99 2.894	3.467 p95 3.47 · p99 3.471	3.363 p95 3.366 · p99 3.367
1,024 in / 128 out	1,024 in / 128 outEnd to end, ms	377.803 p95 378.201 · p99 379.248	466.844 p95 467.43 · p99 468.458	451.438 p95 452.492 · p99 452.735
1,024 in / 128 out	1,024 in / 128 outOutput tokens/s	338.58	274.09	283.39
1,024 in / 1,024 out	1,024 in / 1,024 outTime to first token, ms	11.873 p95 12.075 · p99 12.129	28.418 p95 29.828 · p99 31.568	24.902 p95 25.31 · p99 25.618
1,024 in / 1,024 out	1,024 in / 1,024 outDecode, ms per token	2.918 p95 2.92 · p99 2.921	3.48 p95 3.482 · p99 3.482	3.394 p95 3.396 · p99 3.396
1,024 in / 1,024 out	1,024 in / 1,024 outEnd to end, ms	2,997.493 p95 2,999.416 · p99 2,999.638	3,588.616 p95 3,589.951 · p99 3,590.934	3,496.947 p95 3,498.912 · p99 3,499.245
1,024 in / 1,024 out	1,024 in / 1,024 outOutput tokens/s	341.58	285.36	292.81
4,096 in / 128 out	4,096 in / 128 outTime to first token, ms	43.765 p95 44.012 · p99 44.037	68.2 p95 70.702 · p99 70.882	58.864 p95 61.588 · p99 62.237
4,096 in / 128 out	4,096 in / 128 outDecode, ms per token	3.143 p95 3.146 · p99 3.146	3.541 p95 3.547 · p99 3.547	3.471 p95 3.476 · p99 3.477
4,096 in / 128 out	4,096 in / 128 outEnd to end, ms	443.003 p95 443.272 · p99 443.368	517.873 p95 520.382 · p99 520.529	500.421 p95 502.69 · p99 502.861
4,096 in / 128 out	4,096 in / 128 outOutput tokens/s	288.86	246.79	255.84
4,096 in / 1,024 out	4,096 in / 1,024 outTime to first token, ms	43.942 p95 44.1 · p99 44.177	69.572 p95 71.051 · p99 71.61	60.106 p95 60.78 · p99 62.327
4,096 in / 1,024 out	4,096 in / 1,024 outDecode, ms per token	3.172 p95 3.176 · p99 3.177	3.565 p95 3.569 · p99 3.569	3.491 p95 3.493 · p99 3.494
4,096 in / 1,024 out	4,096 in / 1,024 outEnd to end, ms	3,288.787 p95 3,293.44 · p99 3,294.085	3,717.26 p95 3,720.244 · p99 3,720.817	3,630.023 p95 3,634.02 · p99 3,634.733
4,096 in / 1,024 out	4,096 in / 1,024 outOutput tokens/s	311.31	275.42	282.04

Two H100s (TP2) · Concurrent streamsevery stream sends 512 prompt tokens and generates 128, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
1 stream	1 streamTime to first token, ms	8.038 p95 8.129 · p99 8.131	25.985 p95 28.803 · p99 28.892	13.13 p95 13.603 · p99 13.655
1 stream	1 streamDecode, ms per token	2.839 p95 2.84 · p99 2.84	3.45 p95 3.453 · p99 3.453	3.337 p95 3.339 · p99 3.34
1 stream	1 streamEnd to end, ms	368.503 p95 368.726 · p99 368.781	464.384 p95 466.457 · p99 466.469	436.92 p95 437.323 · p99 437.588
1 stream	1 streamOutput tokens/s	347.23	275.4	292.88
1 stream	1 streamRequests/s	2.713	2.152	2.288
8 streams	8 streamsTime to first token, ms	30.036 p95 43.635 · p99 43.709	53.001 p95 72.713 · p99 82.077	31.442 p95 52.08 · p99 52.579
8 streams	8 streamsDecode, ms per token	3.193 p95 3.312 · p99 3.39	3.816 p95 3.997 · p99 4.088	3.497 p95 3.776 · p99 3.779
8 streams	8 streamsEnd to end, ms	433.986 p95 438.372 · p99 438.57	537.93 p95 548.04 · p99 551.79	495.056 p95 496.174 · p99 496.304
8 streams	8 streamsOutput tokens/s	2,351.77	1,895.02	2,066.57
8 streams	8 streamsRequests/s	18.373	14.805	16.145
32 streams	32 streamsTime to first token, ms	35.773 p95 61.332 · p99 148.827	102.15 p95 107.254 · p99 226.455	102.324 p95 182.316 · p99 183.331
32 streams	32 streamsDecode, ms per token	4.613 p95 4.697 · p99 4.803	5.122 p95 5.507 · p99 5.683	4.636 p95 5.011 · p99 5.018
32 streams	32 streamsEnd to end, ms	622.457 p95 631.104 · p99 712.314	752.336 p95 769.275 · p99 869.189	689.956 p95 691.866 · p99 692.115
32 streams	32 streamsOutput tokens/s	6,596.9	5,423.86	5,927.84
32 streams	32 streamsRequests/s	51.538	42.374	46.311
64 streams	64 streamsTime to first token, ms	39.023 p95 63.604 · p99 274.034	108.477 p95 136.669 · p99 398.696	180.92 p95 300.011 · p99 537.397
64 streams	64 streamsDecode, ms per token	6.371 p95 6.503 · p99 6.577 SGLang leads	7.305 p95 7.708 · p99 7.887	6.172 p95 6.711 · p99 6.779
64 streams	64 streamsEnd to end, ms	852.947 p95 874.798 · p99 1,063.32	1,036.973 p95 1,063.78 · p99 1,304.077	965.038 p95 968.13 · p99 1,023.767
64 streams	64 streamsOutput tokens/s	9,587.64	7,858.08	8,480.7
64 streams	64 streamsRequests/s	74.903	61.391	66.255
128 streams	128 streamsTime to first token, ms	44.528 p95 68.58 · p99 576.179	142.236 p95 205.191 · p99 803.864	219.642 p95 511.014 · p99 569.4
128 streams	128 streamsDecode, ms per token	10.057 p95 10.167 · p99 10.228	11.602 p95 12.03 · p99 12.245	10.277 p95 11.152 · p99 12.533
128 streams	128 streamsEnd to end, ms	1,324.584 p95 1,351.068 · p99 1,824.584	1,615.707 p95 1,661.587 · p99 2,251.762	1,517.402 p95 1,747.004 · p99 1,842.215
128 streams	128 streamsOutput tokens/s	12,307.86	10,071.74	10,682.72
128 streams	128 streamsRequests/s	96.155	78.685	83.459

Two H100s (TP2) · Prefillone prompt, 1 output token, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
4,096-token prompt	4,096-token promptPrompt tokens/s	90,229.87	60,141.9	70,305.79
4,096-token prompt	4,096-token promptTime to first token, ms	45.196 p95 45.702 · p99 45.766	67.848 p95 69.119 · p99 69.366	57.796 p95 59.09 · p99 60.617
16,384-token prompt	16,384-token promptPrompt tokens/s	70,978.23	52,916.46	60,031.58
16,384-token prompt	16,384-token promptTime to first token, ms	229.973 p95 234.488 · p99 234.646	308.992 p95 310.934 · p99 311.128	271.787 p95 276.749 · p99 277.068

Two H100s (TP2) · Agentic loopeach turn re-sends the accumulated conversation and generates 48 tokens
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	143.54 p95 143.606 · p99 144.126	176.437 p95 177.059 · p99 177.593	204.641 p95 205.486 · p99 205.642
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	4.159 p95 4.206 · p99 4.211	12.559 p95 12.968 · p99 13.25	45.357 p95 46.153 · p99 46.329
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	143.394 p95 143.464 · p99 143.99	176.299 p95 176.896 · p99 177.397	204.427 p95 205.282 · p99 205.433
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	169.403 p95 169.521 · p99 169.61	199.587 p95 201.305 · p99 201.924	226.009 p95 227.778 · p99 228.854
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	5.999 p95 6.093 · p99 6.098	28.834 p95 30.691 · p99 31.282	58.73 p95 59.879 · p99 60.595
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	169.173 p95 169.302 · p99 169.367	199.205 p95 200.885 · p99 201.553	225.637 p95 227.375 · p99 228.463
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	724.413 p95 724.89 · p99 725.052	888.67 p95 898.478 · p99 899.148	1,034.626 p95 1,052.954 · p99 1,056.886
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	4.234 p95 4.507 · p99 4.555	13.504 p95 15.971 · p99 16.563	46.83 p95 51.586 · p99 52.583
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	144.136 p95 146.371 · p99 146.462	177.801 p95 180.482 · p99 181.24	206.105 p95 210.827 · p99 212.494
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	848.354 p95 848.783 · p99 848.889	993.241 p95 1,015.806 · p99 1,016.186	1,126.619 p95 1,143.471 · p99 1,144.134
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	6.183 p95 6.328 · p99 6.411	27.984 p95 32.532 · p99 34.816	56.908 p95 60.946 · p99 62.092
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	169.346 p95 170.084 · p99 170.13	198.519 p95 202.682 · p99 204.322	225.381 p95 228.846 · p99 230.206
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,466.007 p95 1,466.463 · p99 1,466.828	1,794.99 p95 1,815.943 · p99 1,818.743	2,068.338 p95 2,091.577 · p99 2,092.054
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	4.46 p95 4.65 · p99 4.745	14.667 p95 17.695 · p99 18.508	45.289 p95 50.351 · p99 51.148
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	146.356 p95 149.562 · p99 149.667	179.303 p95 183.248 · p99 184.64	206.096 p95 211.622 · p99 212.741
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,706.753 p95 1,707.462 · p99 1,707.682	2,008.225 p95 2,033.682 · p99 2,044.724	2,248.313 p95 2,271.384 · p99 2,277.319
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	6.305 p95 6.493 · p99 6.605	29.808 p95 34.473 · p99 35.4	55.727 p95 59.153 · p99 60.645
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	170.294 p95 172.374 · p99 172.482	200.784 p95 205.351 · p99 206.391	224.673 p95 228.156 · p99 229.775

Two H100s (TP2) · Tool loopthe same loop with tool definitions attached and a 160-token budget per turn
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	476.486 p95 476.719 · p99 476.742	570.118 p95 571.199 · p99 571.403	587.11 p95 589.226 · p99 589.259
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	4.249 p95 4.463 · p99 4.472	13.651 p95 15.324 · p99 15.553	45.754 p95 48.012 · p99 48.231
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	476.315 p95 476.502 · p99 476.542	569.905 p95 570.963 · p99 571.161	586.792 p95 588.974 · p99 588.984
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	560.132 p95 560.386 · p99 560.66	612.092 p95 616.375 · p99 616.816	629.651 p95 632.8 · p99 634.019
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	6.05 p95 6.115 · p99 6.17	28.751 p95 32.172 · p99 33.699	58.016 p95 61.363 · p99 62.648
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	559.86 p95 560.106 · p99 560.404	611.78 p95 615.999 · p99 616.449	629.287 p95 632.343 · p99 633.604
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	1,688.317 p95 1,688.944 · p99 1,689.097	2,462.99 p95 2,472.952 · p99 2,473.727	2,563.841 p95 2,574.741 · p99 2,575.303
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	4.44 p95 4.648 · p99 4.712	14.993 p95 17.753 · p99 18.472	46.572 p95 49.871 · p99 51.928
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	475.963 p95 485.635 · p99 485.726	572.075 p95 579.063 · p99 579.508	589.122 p95 598.363 · p99 599.225
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	2,386.008 p95 2,387.108 · p99 2,387.377	2,636.341 p95 2,641.583 · p99 2,642.542	2,730.221 p95 2,740.188 · p99 2,741.328
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	6.396 p95 6.545 · p99 6.818	29.161 p95 31.268 · p99 31.696	58.644 p95 61.629 · p99 62.714
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	558.594 p95 565.547 · p99 565.63	614.224 p95 618.917 · p99 619.852	631.219 p95 638.309 · p99 640.144
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	3,807.784 p95 3,809.704 · p99 3,809.939	5,382.262 p95 5,397.085 · p99 5,398.868	5,554.796 p95 5,578.089 · p99 5,581.829
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	4.576 p95 4.87 · p99 4.947	16.651 p95 21.225 · p99 22.075	47.463 p95 50.828 · p99 52.218
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	479.199 p95 505.493 · p99 505.761	578.837 p95 586.076 · p99 587.854	596.326 p95 602.043 · p99 603.791
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	5,221.297 p95 5,224.386 · p99 5,224.54	5,785.214 p95 5,809.55 · p99 5,814.701	5,938.374 p95 5,958.714 · p99 5,964.971
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	6.471 p95 6.706 · p99 6.752	32.292 p95 37.065 · p99 41.892	60.014 p95 63.771 · p99 64.955
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	562.949 p95 571.941 · p99 572.258	621.444 p95 630.491 · p99 636.306	637.94 p95 646.249 · p99 646.877

Model

Qwen3-30B-A3B

Qwen/Qwen3-30B-A3BTwo H100s (TP2)

Decode latency under concurrency

ms per output token per request, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 9.967, vLLM 19.148, SGLang 11.372 ms/token. Every stream sends 512 prompt tokens and generates 128.

Time to first token under concurrency

ms, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 451.672, vLLM 182.423, SGLang 394.956 ms. SGLang leads at 8 streams. vLLM leads at 64 and 128 streams. Every stream sends 512 prompt tokens and generates 128.

Output throughput under concurrency

generated tokens per second over the window, higher is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 9,534.51, vLLM 6,221.56, SGLang 8,875.55 tok/s. Every stream sends 512 prompt tokens and generates 128.

Prefill throughput

prompt tokens per second, 1 output token, higher is better

LayerScalevLLMSGLang

On the 4,096-token prompt: LayerScale 67,690.95, vLLM 39,881.73, SGLang 59,809.17 tokens/s. On the 16,384-token prompt: LayerScale 55,340.36, vLLM 40,287.58, SGLang 51,555.62 tokens/s.

Single-stream latency

end to end per request, ms, p50, lower is better, prompt tokens → generated tokens

LayerScalevLLMSGLang

128 in / 128 out: LayerScale 363.908, vLLM 515.196, SGLang 523.577 ms. 4,096 in / 1,024 out: LayerScale 3,332.367, vLLM 4,244.763, SGLang 4,350.998 ms. Time to first token and per-token decode for every point are in the table.

Multi-turn agentic loop

whole loop wall time, ms, p50, 48 output tokens per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 1,768.197, vLLM 2,238.995, SGLang 2,769.48 ms.

Multi-turn tool loop

whole loop wall time with tool definitions attached, ms, p50, 160-token budget per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 5,749.38, vLLM 6,930.963, SGLang 7,616.911 ms. vLLM is not shown at 10 turns · 2,048-token prefix: that measurement recorded 20 failed requests and is excluded.

Two H100s (TP2) · Single streamone request at a time, 20 iterations after 3 warm-up, p50 with p95 and p99 beneath
Workload	Metric	LayerScale	vLLM	SGLang
128 in / 128 out	128 in / 128 outTime to first token, ms	11.428 p95 11.734 · p99 11.77	16.356 p95 17.162 · p99 17.653	17.742 p95 18.219 · p99 18.428
128 in / 128 out	128 in / 128 outDecode, ms per token	2.776 p95 2.778 · p99 2.78	3.928 p95 3.93 · p99 3.93	3.982 p95 3.984 · p99 3.985
128 in / 128 out	128 in / 128 outEnd to end, ms	363.908 p95 364.512 · p99 364.698	515.196 p95 515.98 · p99 516.72	523.577 p95 524.004 · p99 524.077
128 in / 128 out	128 in / 128 outOutput tokens/s	351.55	248.32	244.39
128 in / 1,024 out	128 in / 1,024 outTime to first token, ms	11.613 p95 11.748 · p99 11.841	16.892 p95 17.16 · p99 17.504	17.735 p95 18.226 · p99 18.412
128 in / 1,024 out	128 in / 1,024 outDecode, ms per token	2.816 p95 2.819 · p99 2.819	3.936 p95 3.937 · p99 3.937	4.01 p95 4.011 · p99 4.011
128 in / 1,024 out	128 in / 1,024 outEnd to end, ms	2,892.451 p95 2,895.067 · p99 2,895.159	4,043.021 p95 4,044.01 · p99 4,044.627	4,119.652 p95 4,120.553 · p99 4,122.126
128 in / 1,024 out	128 in / 1,024 outOutput tokens/s	353.99	253.25	248.55
1,024 in / 128 out	1,024 in / 128 outTime to first token, ms	20.109 p95 20.358 · p99 20.374	51.475 p95 53.992 · p99 54.29	25.052 p95 25.848 · p99 26.3
1,024 in / 128 out	1,024 in / 128 outDecode, ms per token	2.857 p95 2.859 · p99 2.868	3.954 p95 3.956 · p99 3.956	4.027 p95 4.029 · p99 4.03
1,024 in / 128 out	1,024 in / 128 outEnd to end, ms	383.01 p95 383.305 · p99 384.516	553.423 p95 555.819 · p99 556.324	536.337 p95 537.18 · p99 537.577
1,024 in / 128 out	1,024 in / 128 outOutput tokens/s	334.03	230.95	238.5
1,024 in / 1,024 out	1,024 in / 1,024 outTime to first token, ms	20.331 p95 20.432 · p99 20.499	53.198 p95 54.336 · p99 55.636	25.316 p95 25.959 · p99 26.001
1,024 in / 1,024 out	1,024 in / 1,024 outDecode, ms per token	2.904 p95 2.906 · p99 2.907	3.98 p95 3.98 · p99 3.981	4.058 p95 4.059 · p99 4.059
1,024 in / 1,024 out	1,024 in / 1,024 outEnd to end, ms	2,990.614 p95 2,992.672 · p99 2,993.865	4,124.642 p95 4,125.443 · p99 4,126.278	4,176.768 p95 4,177.274 · p99 4,177.509
1,024 in / 1,024 out	1,024 in / 1,024 outOutput tokens/s	342.38	248.26	245.15
4,096 in / 128 out	4,096 in / 128 outTime to first token, ms	60.455 p95 60.801 · p99 60.865	103.129 p95 105.895 · p99 109.14	67.694 p95 68.639 · p99 68.946
4,096 in / 128 out	4,096 in / 128 outDecode, ms per token	3.17 p95 3.172 · p99 3.175	4.027 p95 4.029 · p99 4.029	4.164 p95 4.167 · p99 4.168
4,096 in / 128 out	4,096 in / 128 outEnd to end, ms	463.072 p95 463.4 · p99 463.597	614.561 p95 617.338 · p99 620.237	596.552 p95 597.459 · p99 597.619
4,096 in / 128 out	4,096 in / 128 outOutput tokens/s	276.28	208.05	214.46
4,096 in / 1,024 out	4,096 in / 1,024 outTime to first token, ms	60.516 p95 60.739 · p99 60.798	105.788 p95 107.126 · p99 108.004	68.687 p95 69.573 · p99 70.116
4,096 in / 1,024 out	4,096 in / 1,024 outDecode, ms per token	3.198 p95 3.2 · p99 3.201	4.046 p95 4.047 · p99 4.047	4.186 p95 4.187 · p99 4.187
4,096 in / 1,024 out	4,096 in / 1,024 outEnd to end, ms	3,332.367 p95 3,334.502 · p99 3,334.511	4,244.763 p95 4,246.494 · p99 4,248.365	4,350.998 p95 4,351.763 · p99 4,353.598
4,096 in / 1,024 out	4,096 in / 1,024 outOutput tokens/s	307.3	241.22	235.34

Two H100s (TP2) · Concurrent streamsevery stream sends 512 prompt tokens and generates 128, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
1 stream	1 streamTime to first token, ms	15.163 p95 15.541 · p99 15.61	50.71 p95 52.849 · p99 53.595	20.84 p95 21.303 · p99 21.438
1 stream	1 streamDecode, ms per token	2.801 p95 2.804 · p99 2.804	3.923 p95 3.925 · p99 3.925	4.002 p95 4.004 · p99 4.004
1 stream	1 streamEnd to end, ms	370.857 p95 371.692 · p99 371.695	549.034 p95 550.639 · p99 551.781	529.087 p95 529.466 · p99 529.786
1 stream	1 streamOutput tokens/s	344.88	233.01	241.84
1 stream	1 streamRequests/s	2.694	1.82	1.889
8 streams	8 streamsTime to first token, ms	49.412 p95 54.907 · p99 55.169 SGLang leads	87.92 p95 95.142 · p99 147.464	47.768 p95 69.938 · p99 70.539
8 streams	8 streamsDecode, ms per token	4.069 p95 4.178 · p99 4.278	6.057 p95 6.334 · p99 6.355	4.682 p95 4.896 · p99 4.899
8 streams	8 streamsEnd to end, ms	557.407 p95 558.791 · p99 566.299	857.168 p95 861.345 · p99 866.962	642.247 p95 643.143 · p99 643.698
8 streams	8 streamsOutput tokens/s	1,834.82	1,198.17	1,593.03
8 streams	8 streamsRequests/s	14.335	9.361	12.446
32 streams	32 streamsTime to first token, ms	125.898 p95 215.232 · p99 225.591	136.474 p95 184.819 · p99 384.03	129.68 p95 229.869 · p99 230.732
32 streams	32 streamsDecode, ms per token	5.334 p95 6.036 · p99 6.179	9.74 p95 10.15 · p99 10.396	6.058 p95 6.663 · p99 6.671
32 streams	32 streamsEnd to end, ms	803.081 p95 804.715 · p99 804.919	1,371.2 p95 1,390.178 · p99 1,469.952	898.808 p95 899.91 · p99 900.256
32 streams	32 streamsOutput tokens/s	5,112.71	3,037.73	4,553.56
32 streams	32 streamsRequests/s	39.943	23.732	35.575
64 streams	64 streamsTime to first token, ms	233.822 p95 426.18 · p99 440.232 vLLM leads	138.036 p95 189.828 · p99 670.646	164.38 p95 412.462 · p99 464.1
64 streams	64 streamsDecode, ms per token	6.867 p95 8.361 · p99 8.555	13.409 p95 14.175 · p99 14.527	8.605 p95 9.447 · p99 9.485
64 streams	64 streamsEnd to end, ms	1,106.429 p95 1,108.257 · p99 1,108.875	1,847.32 p95 1,941.757 · p99 2,234.773	1,257.869 p95 1,278.363 · p99 1,583.364
64 streams	64 streamsOutput tokens/s	7,395.69	4,404.06	6,472.16
64 streams	64 streamsRequests/s	57.779	34.407	50.564
128 streams	128 streamsTime to first token, ms	451.672 p95 829.088 · p99 860.115 vLLM leads	182.423 p95 231.809 · p99 1,329.078	394.956 p95 679.434 · p99 708.303
128 streams	128 streamsDecode, ms per token	9.967 p95 13.033 · p99 13.345	19.148 p95 19.906 · p99 20.009	11.372 p95 12.833 · p99 13.867
128 streams	128 streamsEnd to end, ms	1,716.029 p95 1,717.902 · p99 1,718.37	2,610.521 p95 2,750.867 · p99 3,692.518	1,836.604 p95 2,011.853 · p99 2,204.893
128 streams	128 streamsOutput tokens/s	9,534.51	6,221.56	8,875.55
128 streams	128 streamsRequests/s	74.488	48.606	69.34

Two H100s (TP2) · Prefillone prompt, 1 output token, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
4,096-token prompt	4,096-token promptPrompt tokens/s	67,690.95	39,881.73	59,809.17
4,096-token prompt	4,096-token promptTime to first token, ms	60.228 p95 60.572 · p99 60.693	102.103 p95 104.755 · p99 105.158	68.174 p95 69.919 · p99 70.304
16,384-token prompt	16,384-token promptPrompt tokens/s	55,340.36	40,287.58	51,555.62
16,384-token prompt	16,384-token promptTime to first token, ms	295.398 p95 298.211 · p99 298.407	403.84 p95 411.557 · p99 448.306	317.221 p95 320.807 · p99 320.934

Two H100s (TP2) · Agentic loopeach turn re-sends the accumulated conversation and generates 48 tokens
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	146.788 p95 146.897 · p99 146.933	205.363 p95 206.097 · p99 206.367	255.144 p95 256.719 · p99 262.639
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	8.463 p95 8.545 · p99 8.561	18.185 p95 19.034 · p99 19.471	64.334 p95 65.949 · p99 71.866
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	146.639 p95 146.762 · p99 146.764	205.091 p95 205.801 · p99 206.049	254.975 p95 256.543 · p99 262.448
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	176.088 p95 176.313 · p99 176.329	221.457 p95 223.297 · p99 223.887	276.97 p95 279.669 · p99 279.976
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	10.255 p95 10.515 · p99 10.565	29.939 p95 31.47 · p99 32.427	78.95 p95 81.685 · p99 82.055
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	175.855 p95 176.06 · p99 176.086	221.146 p95 222.94 · p99 223.554	276.722 p95 279.355 · p99 279.686
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	735.053 p95 735.569 · p99 735.651	1,018.666 p95 1,028.325 · p99 1,028.612	1,284.318 p95 1,300.646 · p99 1,302.785
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	5.985 p95 8.701 · p99 8.741	16.398 p95 18.886 · p99 19.47	65.702 p95 69.657 · p99 71.739
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	147.217 p95 148.195 · p99 148.229	204.045 p95 206.476 · p99 206.813	257.211 p95 260.829 · p99 263.114
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	879.651 p95 881.647 · p99 882.14	1,113.465 p95 1,127.075 · p99 1,127.84	1,380.21 p95 1,397.324 · p99 1,400.976
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	10.361 p95 10.858 · p99 10.969	30.849 p95 34.656 · p99 34.856	77.154 p95 81.664 · p99 83.355
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	175.665 p95 176.47 · p99 176.71	222.556 p95 225.466 · p99 225.624	275.831 p95 279.807 · p99 280.785
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	1,489.762 p95 1,490.997 · p99 1,491.591	2,048.206 p95 2,072.872 · p99 2,073.787	2,588.573 p95 2,613.379 · p99 2,618.625
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	6.052 p95 8.799 · p99 8.84	16.416 p95 19.127 · p99 19.807	66.785 p95 70.571 · p99 71.419
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	148.538 p95 152.131 · p99 152.217	205.089 p95 208.3 · p99 209.495	259.064 p95 263.356 · p99 264.565
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	1,768.197 p95 1,771.521 · p99 1,772.125	2,238.995 p95 2,265.831 · p99 2,268.064	2,769.48 p95 2,799.202 · p99 2,811.607
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	10.706 p95 11.152 · p99 11.29	32.418 p95 36.057 · p99 36.605	77.779 p95 82.66 · p99 83.682
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	176.375 p95 178.863 · p99 179.188	223.975 p95 227.199 · p99 228.034	277.151 p95 282.021 · p99 283.169

Two H100s (TP2) · Tool loopthe same loop with tool definitions attached and a 160-token budget per turn
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	478.336 p95 478.693 · p99 478.873	653.387 p95 654.873 · p99 655.043	715.728 p95 718.301 · p99 718.356
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	8.759 p95 8.957 · p99 9.055	17.42 p95 19.151 · p99 19.29	68.102 p95 70.819 · p99 70.908
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	478.177 p95 478.504 · p99 478.706	653.151 p95 654.571 · p99 654.759	715.463 p95 717.988 · p99 718.066
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	572.554 p95 572.783 · p99 572.819	687.204 p95 689.471 · p99 690.218	753.153 p95 756.277 · p99 756.622
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	10.372 p95 10.659 · p99 10.682	32.636 p95 35.16 · p99 35.32	79.667 p95 83.14 · p99 83.183
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	572.278 p95 572.478 · p99 572.482	686.733 p95 688.946 · p99 689.646	752.798 p95 755.887 · p99 756.224
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	2,416.583 p95 2,418.312 · p99 2,424.45	3,278.758 p95 3,281.634 · p99 3,289.218	3,599.141 p95 3,606.506 · p99 3,610.81
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	6.145 p95 8.955 · p99 9.107	16.734 p95 18.174 · p99 18.902	68.35 p95 71.376 · p99 72.183
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	483.408 p95 488.93 · p99 489.131	654.928 p95 659.984 · p99 661.416	719.684 p95 724.623 · p99 726.818
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	2,867.646 p95 2,869.826 · p99 2,870.264	3,451.203 p95 3,457.344 · p99 3,458.09	3,786.824 p95 3,798.665 · p99 3,800.963
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	10.676 p95 11.246 · p99 11.301	33.314 p95 35.901 · p99 36.245	81.83 p95 85.244 · p99 85.583
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	572.039 p95 580.592 · p99 580.815	689.601 p95 693.507 · p99 693.904	756.759 p95 761.995 · p99 763.555
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	4,958.68 p95 4,960.138 · p99 4,960.226	excluded measurement failed, see note	7,270.931 p95 7,280.35 · p99 7,281.257
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	8.636 p95 8.931 · p99 9.036	excluded measurement failed, see note	70.681 p95 74.299 · p99 75.311
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	489.497 p95 515.3 · p99 515.515	excluded measurement failed, see note	726.509 p95 737.508 · p99 739.932
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	5,749.38 p95 5,751.313 · p99 5,752.129	6,930.963 p95 6,949.383 · p99 6,953.508	7,616.911 p95 7,632.428 · p99 7,634.25
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	10.991 p95 11.451 · p99 11.513	34.63 p95 37.758 · p99 39.753	82.731 p95 87.019 · p99 88.141
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	575.177 p95 579.884 · p99 580.238	693.012 p95 698.845 · p99 700.667	761.682 p95 769.794 · p99 771.716

Model

Qwen3-32B

Qwen/Qwen3-32BTwo H100s (TP2)

Decode latency under concurrency

ms per output token per request, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 44.251, vLLM 45.212, SGLang 34.886 ms/token. SGLang leads at 32, 64 and 128 streams. Every stream sends 512 prompt tokens and generates 128.

Time to first token under concurrency

ms, p50, lower is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 237.74, vLLM 366.413, SGLang 2,026.413 ms. Every stream sends 512 prompt tokens and generates 128.

Output throughput under concurrency

generated tokens per second over the window, higher is better

LayerScalevLLMSGLang

At 128 streams: LayerScale 2,777.69, vLLM 2,662.53, SGLang 2,547.14 tok/s. Every stream sends 512 prompt tokens and generates 128.

Prefill throughput

prompt tokens per second, 1 output token, higher is better

LayerScalevLLMSGLang

On the 4,096-token prompt: LayerScale 16,614.92, vLLM 15,474.19, SGLang 15,462.22 tokens/s. On the 16,384-token prompt: LayerScale 14,077.57, vLLM 13,367.19, SGLang 13,431.38 tokens/s.

Single-stream latency

end to end per request, ms, p50, lower is better, prompt tokens → generated tokens

LayerScalevLLMSGLang

128 in / 128 out: LayerScale 1,618.197, vLLM 1,790.59, SGLang 1,770.032 ms. 4,096 in / 1,024 out: LayerScale 13,719.383, vLLM 14,733.697, SGLang 14,536.334 ms. Time to first token and per-token decode for every point are in the table.

Multi-turn agentic loop

whole loop wall time, ms, p50, 48 output tokens per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 6,476.736, vLLM 7,132.093, SGLang 7,246.33 ms.

Multi-turn tool loop

whole loop wall time with tool definitions attached, ms, p50, 160-token budget per turn, lower is better

LayerScalevLLMSGLang

10 turns · 8,192-token prefix: LayerScale 21,519.568, vLLM 23,309.492, SGLang 23,230.139 ms.

Two H100s (TP2) · Single streamone request at a time, 20 iterations after 3 warm-up, p50 with p95 and p99 beneath
Workload	Metric	LayerScale	vLLM	SGLang
128 in / 128 out	128 in / 128 outTime to first token, ms	16.859 p95 16.996 · p99 17.091	22.963 p95 23.272 · p99 23.313	33.16 p95 33.472 · p99 33.485
128 in / 128 out	128 in / 128 outDecode, ms per token	12.609 p95 12.617 · p99 12.617	13.917 p95 13.933 · p99 13.934	13.678 p95 13.684 · p99 13.688
128 in / 128 out	128 in / 128 outEnd to end, ms	1,618.197 p95 1,619.218 · p99 1,619.323	1,790.59 p95 1,792.506 · p99 1,792.82	1,770.032 p95 1,771.282 · p99 1,771.337
128 in / 128 out	128 in / 128 outOutput tokens/s	79.1	71.47	72.3
128 in / 1,024 out	128 in / 1,024 outTime to first token, ms	17.037 p95 17.233 · p99 17.264	22.89 p95 23.651 · p99 23.938	33.349 p95 33.791 · p99 33.793
128 in / 1,024 out	128 in / 1,024 outDecode, ms per token	12.675 p95 12.678 · p99 12.678	13.962 p95 13.965 · p99 13.966	13.723 p95 13.726 · p99 13.726
128 in / 1,024 out	128 in / 1,024 outEnd to end, ms	12,983.883 p95 12,986.376 · p99 12,986.591	14,306.63 p95 14,309.538 · p99 14,310.082	14,072.117 p95 14,074.794 · p99 14,074.958
128 in / 1,024 out	128 in / 1,024 outOutput tokens/s	78.87	71.57	72.77
1,024 in / 128 out	1,024 in / 128 outTime to first token, ms	55.474 p95 56.85 · p99 57.469	65.25 p95 65.762 · p99 65.802	71.007 p95 71.488 · p99 71.605
1,024 in / 128 out	1,024 in / 128 outDecode, ms per token	12.748 p95 12.761 · p99 12.763	14.001 p95 14.009 · p99 14.009	13.767 p95 13.777 · p99 13.778
1,024 in / 128 out	1,024 in / 128 outEnd to end, ms	1,674.919 p95 1,676.853 · p99 1,677.024	1,843.316 p95 1,844.326 · p99 1,844.998	1,819.325 p95 1,820.749 · p99 1,820.843
1,024 in / 128 out	1,024 in / 128 outOutput tokens/s	76.42	69.44	70.35
1,024 in / 1,024 out	1,024 in / 1,024 outTime to first token, ms	56.076 p95 56.754 · p99 56.788	65.257 p95 66.42 · p99 66.572	70.898 p95 71.473 · p99 71.802
1,024 in / 1,024 out	1,024 in / 1,024 outDecode, ms per token	12.802 p95 12.809 · p99 12.81	14.017 p95 14.02 · p99 14.021	13.795 p95 13.797 · p99 13.798
1,024 in / 1,024 out	1,024 in / 1,024 outEnd to end, ms	13,153.148 p95 13,160.2 · p99 13,160.584	14,404.313 p95 14,408.01 · p99 14,408.389	14,183.652 p95 14,185.261 · p99 14,185.275
1,024 in / 1,024 out	1,024 in / 1,024 outOutput tokens/s	77.85	71.09	72.2
4,096 in / 128 out	4,096 in / 128 outTime to first token, ms	235.068 p95 238.169 · p99 239.641	253.3 p95 254.929 · p99 255.882	253.676 p95 255.163 · p99 255.62
4,096 in / 128 out	4,096 in / 128 outDecode, ms per token	13.176 p95 13.182 · p99 13.183	14.148 p95 14.16 · p99 14.162	13.943 p95 13.95 · p99 13.951
4,096 in / 128 out	4,096 in / 128 outEnd to end, ms	1,908.622 p95 1,912.32 · p99 1,912.791	2,049.984 p95 2,052.784 · p99 2,053.861	2,024.312 p95 2,026.382 · p99 2,026.761
4,096 in / 128 out	4,096 in / 128 outOutput tokens/s	67.05	62.42	63.22
4,096 in / 1,024 out	4,096 in / 1,024 outTime to first token, ms	235.681 p95 238.193 · p99 238.546	253.344 p95 257.163 · p99 258.451	253.898 p95 254.936 · p99 256.659
4,096 in / 1,024 out	4,096 in / 1,024 outDecode, ms per token	13.181 p95 13.181 · p99 13.181	14.154 p95 14.156 · p99 14.157	13.961 p95 13.964 · p99 13.964
4,096 in / 1,024 out	4,096 in / 1,024 outEnd to end, ms	13,719.383 p95 13,722.523 · p99 13,723.048	14,733.697 p95 14,737.083 · p99 14,738.8	14,536.334 p95 14,539.361 · p99 14,539.553
4,096 in / 1,024 out	4,096 in / 1,024 outOutput tokens/s	74.64	69.5	70.44

Two H100s (TP2) · Concurrent streamsevery stream sends 512 prompt tokens and generates 128, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
1 stream	1 streamTime to first token, ms	30.635 p95 30.711 · p99 30.774	37.518 p95 37.928 · p99 38.362	46.362 p95 46.861 · p99 47.133
1 stream	1 streamDecode, ms per token	12.673 p95 12.683 · p99 12.683	13.951 p95 13.966 · p99 13.969	13.717 p95 13.728 · p99 13.731
1 stream	1 streamEnd to end, ms	1,640.218 p95 1,641.351 · p99 1,641.36	1,809.41 p95 1,811.046 · p99 1,811.574	1,788.548 p95 1,789.909 · p99 1,789.967
1 stream	1 streamOutput tokens/s	78.04	70.73	71.56
1 stream	1 streamRequests/s	0.61	0.553	0.559
8 streams	8 streamsTime to first token, ms	143.841 p95 223.589 · p99 233.646	166.258 p95 208.643 · p99 255.092	211.912 p95 242.549 · p99 243.012
8 streams	8 streamsDecode, ms per token	14.288 p95 14.468 · p99 14.476	14.782 p95 15.767 · p99 15.834	14.516 p95 14.763 · p99 14.766
8 streams	8 streamsEnd to end, ms	1,943.798 p95 1,948.822 · p99 1,958.541	2,062.895 p95 2,072.708 · p99 2,111.287	2,083.952 p95 2,085.811 · p99 2,085.924
8 streams	8 streamsOutput tokens/s	526.37	495.81	491.22
8 streams	8 streamsRequests/s	4.112	3.874	3.838
32 streams	32 streamsTime to first token, ms	179.719 p95 345.738 · p99 831.476	267.732 p95 461.967 · p99 883.078	663.066 p95 957.329 · p99 958.238
32 streams	32 streamsDecode, ms per token	19.753 p95 20.229 · p99 20.806 SGLang leads	20.195 p95 21.299 · p99 21.844	17.953 p95 21.527 · p99 21.543
32 streams	32 streamsEnd to end, ms	2,690.229 p95 2,784.979 · p99 3,293.207	2,835.506 p95 2,933.861 · p99 3,343.212	2,942.933 p95 2,945.412 · p99 2,946.092
32 streams	32 streamsOutput tokens/s	1,514.6	1,439.26	1,391.44
32 streams	32 streamsRequests/s	11.833	11.244	10.871
64 streams	64 streamsTime to first token, ms	210.448 p95 362.852 · p99 1,531.738	309.272 p95 499.274 · p99 1,637.404	1,103.21 p95 1,849.341 · p99 1,856.114
64 streams	64 streamsDecode, ms per token	27.46 p95 28.181 · p99 28.797 SGLang leads	28.351 p95 29.617 · p99 30.334	23.663 p95 30.619 · p99 30.674
64 streams	64 streamsEnd to end, ms	3,707.786 p95 3,816.542 · p99 4,980.839	3,923.3 p95 4,034.281 · p99 5,182.766	4,103.376 p95 4,111.353 · p99 4,445.52
64 streams	64 streamsOutput tokens/s	2,197.37	2,077.61	1,996.12
64 streams	64 streamsRequests/s	17.167	16.231	15.595
128 streams	128 streamsTime to first token, ms	237.74 p95 364.762 · p99 3,190.143	366.413 p95 494.902 · p99 3,348.937	2,026.413 p95 3,610.538 · p99 3,651.588
128 streams	128 streamsDecode, ms per token	44.251 p95 44.827 · p99 45.226 SGLang leads	45.212 p95 46.253 · p99 47.001	34.886 p95 47.987 · p99 48.815
128 streams	128 streamsEnd to end, ms	5,868.815 p95 5,998.955 · p99 8,684.74	6,117.619 p95 6,243.983 · p99 8,991.323	6,431.06 p95 6,442.391 · p99 6,771.881
128 streams	128 streamsOutput tokens/s	2,777.69	2,662.53	2,547.14
128 streams	128 streamsRequests/s	21.701	20.801	19.899

Two H100s (TP2) · Prefillone prompt, 1 output token, 20 iterations after 3 warm-up
Workload	Metric	LayerScale	vLLM	SGLang
4,096-token prompt	4,096-token promptPrompt tokens/s	16,614.92	15,474.19	15,462.22
4,096-token prompt	4,096-token promptTime to first token, ms	246.242 p95 248.44 · p99 250.092	264.321 p95 268.98 · p99 269.249	264.876 p95 267.342 · p99 269.605
16,384-token prompt	16,384-token promptPrompt tokens/s	14,077.57	13,367.19	13,431.38
16,384-token prompt	16,384-token promptTime to first token, ms	1,164.222 p95 1,167.927 · p99 1,170.292	1,224.719 p95 1,229.277 · p99 1,229.96	1,219.787 p95 1,222.199 · p99 1,226.211

Two H100s (TP2) · Agentic loopeach turn re-sends the accumulated conversation and generates 48 tokens
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	621.088 p95 621.646 · p99 621.65	684.559 p95 685.67 · p99 686.119	702.929 p95 704.215 · p99 704.241
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	15.789 p95 15.947 · p99 16.009	25.419 p95 25.981 · p99 26.562	53.2 p95 54.006 · p99 54.011
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	620.836 p95 621.382 · p99 621.387	684.3 p95 685.332 · p99 685.826	702.673 p95 703.92 · p99 703.944
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	645.015 p95 645.495 · p99 645.564	709.31 p95 712.806 · p99 712.913	720.554 p95 721.283 · p99 722.485
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	17.791 p95 18.177 · p99 18.25	39.303 p95 41.845 · p99 41.917	55.84 p95 56.844 · p99 57.396
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	644.595 p95 645.093 · p99 645.118	708.909 p95 712.403 · p99 712.488	720.192 p95 720.889 · p99 722.08
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	3,114.601 p95 3,115.934 · p99 3,116.048	3,427.555 p95 3,430.914 · p99 3,436.958	3,521.047 p95 3,523.932 · p99 3,527.873
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	15.727 p95 16.109 · p99 16.138	25.71 p95 27.269 · p99 28.23	53.21 p95 54.13 · p99 54.422
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	621.819 p95 625.215 · p99 625.41	685.026 p95 687.386 · p99 688.648	704.05 p95 705.569 · p99 705.951
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	3,233.003 p95 3,235.299 · p99 3,235.686	3,554.671 p95 3,558.56 · p99 3,560.721	3,607.984 p95 3,614.999 · p99 3,618.048
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	18.05 p95 18.429 · p99 18.478	39.786 p95 41.484 · p99 42.226	56.201 p95 57.735 · p99 59.611
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	646.194 p95 647.575 · p99 647.656	710.131 p95 713.086 · p99 713.511	721.485 p95 723.509 · p99 724.878
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	6,259.604 p95 6,263.516 · p99 6,264.447	6,879.318 p95 6,908.217 · p99 6,909.047	7,053.072 p95 7,057.636 · p99 7,061.108
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	15.729 p95 16.036 · p99 16.146	26.776 p95 30.001 · p99 34.724	52.778 p95 53.896 · p99 54.88
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	625.177 p95 632.261 · p99 632.541	687.825 p95 692.738 · p99 697.003	704.976 p95 707.583 · p99 708.657
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	6,476.736 p95 6,479.159 · p99 6,479.461	7,132.093 p95 7,152.595 · p99 7,154.464	7,246.33 p95 7,251.909 · p99 7,252.201
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	18.209 p95 18.556 · p99 18.689	40.927 p95 44.054 · p99 45.205	58.068 p95 60.498 · p99 61.509
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	647.328 p95 649.292 · p99 649.559	713.15 p95 717.294 · p99 719.471	724.026 p95 728.134 · p99 729.737

Two H100s (TP2) · Tool loopthe same loop with tool definitions attached and a 160-token budget per turn
Workload	Metric	LayerScale	vLLM	SGLang
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixWhole loop, ms	2,064.027 p95 2,064.971 · p99 2,065.218	2,258.241 p95 2,260.747 · p99 2,261.445	2,253.169 p95 2,254.622 · p99 2,254.682
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixTime to first token per turn, ms	15.967 p95 16.051 · p99 16.104	26.668 p95 27.434 · p99 27.458	53.797 p95 54.328 · p99 54.538
1 turn · 2,048-token prefix	1 turn · 2,048-token prefixPer turn end to end, ms	2,063.716 p95 2,064.642 · p99 2,064.909	2,257.943 p95 2,260.417 · p99 2,261.134	2,252.844 p95 2,254.289 · p99 2,254.367
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixWhole loop, ms	2,140.538 p95 2,141.217 · p99 2,141.734	2,315.154 p95 2,316.582 · p99 2,316.723	2,309.658 p95 2,311.492 · p99 2,312.152
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixTime to first token per turn, ms	18.016 p95 18.118 · p99 18.131	41.428 p95 42.912 · p99 43.166	60.307 p95 61.646 · p99 61.677
1 turn · 8,192-token prefix	1 turn · 8,192-token prefixPer turn end to end, ms	2,140.099 p95 2,140.758 · p99 2,141.277	2,314.775 p95 2,316.148 · p99 2,316.259	2,309.223 p95 2,311.064 · p99 2,311.703
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixWhole loop, ms	10,379.188 p95 10,379.86 · p99 10,380.095	11,317.886 p95 11,320.697 · p99 11,321.508	11,297.818 p95 11,299.737 · p99 11,301.864
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixTime to first token per turn, ms	15.862 p95 16.187 · p99 16.209	27.454 p95 29.242 · p99 30.233	53.562 p95 54.488 · p99 54.76
5 turns · 2,048-token prefix	5 turns · 2,048-token prefixPer turn end to end, ms	2,074.633 p95 2,090.461 · p99 2,090.796	2,262.275 p95 2,271.174 · p99 2,272.242	2,259.352 p95 2,266.355 · p99 2,267.03
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixWhole loop, ms	10,730.729 p95 10,731.904 · p99 10,733.088	11,614.144 p95 11,617.092 · p99 11,617.367	11,578.27 p95 11,600.221 · p99 11,607.629
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixTime to first token per turn, ms	18.283 p95 18.657 · p99 18.751	43.578 p95 46.125 · p99 46.93	61.713 p95 65.956 · p99 75.173
5 turns · 8,192-token prefix	5 turns · 8,192-token prefixPer turn end to end, ms	2,146.105 p95 2,151.309 · p99 2,151.673	2,321.27 p95 2,331.89 · p99 2,332.168	2,315.075 p95 2,323.287 · p99 2,327.762
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixWhole loop, ms	20,916.379 p95 20,918.078 · p99 20,918.424	22,691.764 p95 22,701.683 · p99 22,701.752	22,650.839 p95 22,667.965 · p99 22,711.626
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixTime to first token per turn, ms	15.979 p95 16.38 · p99 16.446	28.96 p95 31.772 · p99 32.447	54.107 p95 56.119 · p99 73.408
10 turns · 2,048-token prefix	10 turns · 2,048-token prefixPer turn end to end, ms	2,091.111 p95 2,113.827 · p99 2,114.676	2,271.584 p95 2,277.071 · p99 2,279.4	2,266.41 p95 2,274.862 · p99 2,283.81
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixWhole loop, ms	21,519.568 p95 21,520.58 · p99 21,521.063	23,309.492 p95 23,318.517 · p99 23,319.073	23,230.139 p95 23,254.113 · p99 23,289.553
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixTime to first token per turn, ms	18.511 p95 18.847 · p99 18.964	45.681 p95 49.238 · p99 49.96	63.215 p95 68.264 · p99 78.444
10 turns · 8,192-token prefix	10 turns · 8,192-token prefixPer turn end to end, ms	2,151.432 p95 2,162.35 · p99 2,163.072	2,330.46 p95 2,345.93 · p99 2,347.108	2,322.891 p95 2,336.369 · p99 2,344.046

Method

What was run, and what was left out

Hardware and software. NVIDIA H100 80GB HBM3, two GPUs, driver 580.105.08, CUDA 13.3, clocks locked. The latest LayerScale version as of 5 September 2026, vLLM 0.27.1, SGLang 0.5.17. Every point is 20 iterations after 3 warm-up iterations; p50, p95 and p99 are over those iterations, and the per-stream and per-turn statistics count every request inside them.
Same checkpoint, same settings. The model identifier in every result file is identical across the three engines for a given section, temperature is 0 everywhere, and the chat-template date is pinned so that every engine renders the same prompt.
Prefix caching. Off for the single, concurrent and prefill workloads on all three engines, because those send one calibrated prompt repeatedly and with caching on they would measure a cache lookup rather than the engine. On for the agentic and tool loops on all three engines, because a growing conversation prefix is the workload there.
Failed measurements are excluded, never charted. Every result file records the number of failed requests and any errors. The following measurements recorded failures and are excluded: Qwen3-30B-A3B · tools · t10-p2048 · vLLM (requests.failed=20, errors=2). A missing competitor bar is a missing measurement, not a LayerScale win.
The tool loop is a latency workload. Whether a model chooses to emit a tool call inside its per-turn budget is a property of the model, and it is the same on every engine, so it is not reported as an engine result. What is reported is how long the loop takes.
Chart labels are rounded for display. The value labels drawn on the charts are rounded to a few significant digits so they fit; the tables carry every value at the precision stored in the result file.
Nothing is averaged across models. Different models have different shapes. Every chart and every table is one model, and every ratio on this page names both values it was computed from.