Original GPU benchmarking

Llama 3.1 8B inference GPU comparison

Measured throughput and latency across RunPod, Lambda Cloud and Vast.ai for six exact workloads.

These figures are benchmark-time observations, not current GPU offers or customer savings estimates.

Research scope: Three measured runs per result, recorded 7–9 October 2026. These tests use synthetic random-token prompts, not customer traffic. Prices are historical; latency figures are observations, not production guarantees. Sample sizes and repeat ranges accompany each workload.

What to look for

A40 vs H100: read the focused cost and latency comparison.

Rows are ordered by historical output-token compute cost, lowest first. A lower cost does not establish that a deployment meets your latency requirements.

Evidence types: Standard protocol includes a stabilization phase. Reviewed historical results were accepted after a separate audit, not rerun under the standard protocol. Both labels reflect internal review, not independent certification. How we review the evidence.

Short generation — concurrency 8

1,000 input tokens, 250 output tokens, 8 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 64 completed requests per run (192 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 1000 input and 250 output tokens at concurrency 8
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $0.5975 1,097.2 105.0 7.1 7.3
RunPod / A100 SXM Standard protocol $0.6417 688.3 98.3 11.5 12.2
RunPod / A40 Standard protocol $0.6594 248.5 164.1 31.8 32.8
RunPod / H100 SXM Standard protocol $0.8529 1,136.6 137.4 6.7 7.4
Lambda Cloud / H100 SXM Standard protocol $1.0375 1,148.6 32.9 6.9 7.0

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $0.5975 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
1,097.2
Worst p95 TTFT (ms)
105.0
Worst p95 TPOT (ms)
7.1
Worst p95 ITL (ms)
7.3
p95 TTFT across runs (ms)
92.4–105.0
RunPod / A100 SXM — $0.6417 / 1M output
Evidence type
Standard protocol
Output tokens / sec
688.3
Worst p95 TTFT (ms)
98.3
Worst p95 TPOT (ms)
11.5
Worst p95 ITL (ms)
12.2
p95 TTFT across runs (ms)
86.7–98.3
RunPod / A40 — $0.6594 / 1M output
Evidence type
Standard protocol
Output tokens / sec
248.5
Worst p95 TTFT (ms)
164.1
Worst p95 TPOT (ms)
31.8
Worst p95 ITL (ms)
32.8
p95 TTFT across runs (ms)
133.4–164.1
RunPod / H100 SXM — $0.8529 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,136.6
Worst p95 TTFT (ms)
137.4
Worst p95 TPOT (ms)
6.7
Worst p95 ITL (ms)
7.4
p95 TTFT across runs (ms)
116.6–137.4
Lambda Cloud / H100 SXM — $1.0375 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,148.6
Worst p95 TTFT (ms)
32.9
Worst p95 TPOT (ms)
6.9
Worst p95 ITL (ms)
7.0
p95 TTFT across runs (ms)
30.3–32.9
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
1,096.8–1,100.5
Run-level p95 TTFT range (ms)
92.4–105.0
Run-level p95 TPOT range (ms)
7.0–7.1
Run-level p95 ITL range (ms)
7.3–7.3

RunPod / A100 SXM

Output tokens / sec across runs
687.9–689.6
Run-level p95 TTFT range (ms)
86.7–98.3
Run-level p95 TPOT range (ms)
11.4–11.5
Run-level p95 ITL range (ms)
12.0–12.2

RunPod / A40

Output tokens / sec across runs
248.3–249.1
Run-level p95 TTFT range (ms)
133.4–164.1
Run-level p95 TPOT range (ms)
31.8–31.8
Run-level p95 ITL range (ms)
32.4–32.8

RunPod / H100 SXM

Output tokens / sec across runs
1,134.2–1,159.0
Run-level p95 TTFT range (ms)
116.6–137.4
Run-level p95 TPOT range (ms)
6.5–6.7
Run-level p95 ITL range (ms)
7.3–7.4

Lambda Cloud / H100 SXM

Output tokens / sec across runs
1,148.4–1,148.8
Run-level p95 TTFT range (ms)
30.3–32.9
Run-level p95 TPOT range (ms)
6.9–6.9
Run-level p95 ITL range (ms)
7.0–7.0

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 15:38:41
Measurement finished (UTC)
2026-10-08 15:39:57
Measured repetition duration (seconds)
14.54–14.59
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 0243016d64214ad583c9de3c704b91e507890eb4c7fcf9770da1147668069440
  • 1008eafc5c61f15236e68958d97a06a7db14892da3a11d2c0a0face934824855
  • b46dd1272801a7c3615b81312613a26022500cfd1b1b7599d5733c4ee2289b85

Retained server-log SHA-256 identifier:

076ecae71a431a44e15e41eba444a44d456e911db17bde7137b48d9217de3fa1

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 10:15:01
Measurement finished (UTC)
2026-10-07 10:16:56
Measured repetition duration (seconds)
23.20–23.26
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 88d9c7ad0ad761138bdd001c256ce969248b233669db66a6bc96bc8df729443f
  • d34ca0b18eb803608138f17327b10dd3565ce0c448b05dfb5b7b6655994bc5bc
  • fb641accc4e0f34bee3242c13a6da0df6bc37ad79e6aad9e1a0381c558fad433

Retained server-log SHA-256 identifier:

6aaac55ca772f970ed22038083ba3cde4e9ffc03c3edb5055b95d00f4d6b19bb

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 08:48:42
Measurement finished (UTC)
2026-10-09 08:52:52
Measured repetition duration (seconds)
64.24–64.44
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 2e7db5313b63605da73a8fee136f004d33a260ea4d59ba127982e3bc25742788
  • 923ca8bf64fc008fc99ef057c437544cfd50a785de87ecdd179ecf9faa048f3a
  • d620e77a4814b61e7fe0842faaf0f74ce0d90afd31ff0862c141a75efb68d300

Retained server-log SHA-256 identifier:

9365da7409284af289c728d481504a2f7214a4b86f345364999e932ef6f51228

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:29:10
Measurement finished (UTC)
2026-10-07 11:30:26
Measured repetition duration (seconds)
13.80–14.11
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 0f18656048e4686da7c8a7e5e86723539c462f3e3b25d6c470469dae1cdc441a
  • 2911ddfa66de6283db076e1d2a38e85262e67bea0698c838f33d4e3665a50476
  • eec7937e12afbfac7b26d82ec20b2b9c46a31fca7dddea4b18eb3017fdda85b3

Retained server-log SHA-256 identifier:

48051e78f014738de60f410e08f3014a7e3e3f2c5cc9d45e39459eaa664b5c52

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:36:28
Measurement finished (UTC)
2026-10-08 13:37:37
Measured repetition duration (seconds)
13.93–13.93
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 8196e55b8909d3a39283c76329bd989bbefd303d5f96197b5e65bd86369044a9
  • 99c2fc4a7074f441871d4f29dca64c2e3e769e30e3fd92afda3c16dc1d38f06a
  • ebacf5d2940384cf0978212b0843e6e6ca648195c713f1f861cb84ebc6f8f208

Retained server-log SHA-256 identifier:

7b8b99081e6adb3d6bca673ea4d142e9217f098b3ca26fe48aa4ed9c6a357a93

Short generation — concurrency 32

1,000 input tokens, 250 output tokens, 32 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 128 completed requests per run (384 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 1000 input and 250 output tokens at concurrency 32
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $0.1865 3,515.7 245.8 8.6 8.9
RunPod / A40 Standard protocol $0.2016 813.0 314.8 38.7 40.9
RunPod / A100 SXM Standard protocol $0.2227 1,983.3 188.4 15.7 17.7
RunPod / H100 SXM Standard protocol $0.2609 3,715.7 262.4 8.1 9.4
Lambda Cloud / H100 SXM Standard protocol $0.3139 3,795.8 85.5 8.2 8.4

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $0.1865 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
3,515.7
Worst p95 TTFT (ms)
245.8
Worst p95 TPOT (ms)
8.6
Worst p95 ITL (ms)
8.9
p95 TTFT across runs (ms)
196.5–245.8
RunPod / A40 — $0.2016 / 1M output
Evidence type
Standard protocol
Output tokens / sec
813.0
Worst p95 TTFT (ms)
314.8
Worst p95 TPOT (ms)
38.7
Worst p95 ITL (ms)
40.9
p95 TTFT across runs (ms)
309.2–314.8
RunPod / A100 SXM — $0.2227 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,983.3
Worst p95 TTFT (ms)
188.4
Worst p95 TPOT (ms)
15.7
Worst p95 ITL (ms)
17.7
p95 TTFT across runs (ms)
162.6–188.4
RunPod / H100 SXM — $0.2609 / 1M output
Evidence type
Standard protocol
Output tokens / sec
3,715.7
Worst p95 TTFT (ms)
262.4
Worst p95 TPOT (ms)
8.1
Worst p95 ITL (ms)
9.4
p95 TTFT across runs (ms)
261.2–262.4
Lambda Cloud / H100 SXM — $0.3139 / 1M output
Evidence type
Standard protocol
Output tokens / sec
3,795.8
Worst p95 TTFT (ms)
85.5
Worst p95 TPOT (ms)
8.2
Worst p95 ITL (ms)
8.4
p95 TTFT across runs (ms)
83.6–85.5
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
3,503.2–3,527.2
Run-level p95 TTFT range (ms)
196.5–245.8
Run-level p95 TPOT range (ms)
8.5–8.6
Run-level p95 ITL range (ms)
8.8–8.9

RunPod / A40

Output tokens / sec across runs
811.8–813.5
Run-level p95 TTFT range (ms)
309.2–314.8
Run-level p95 TPOT range (ms)
38.6–38.7
Run-level p95 ITL range (ms)
40.1–40.9

RunPod / A100 SXM

Output tokens / sec across runs
1,975.0–1,984.2
Run-level p95 TTFT range (ms)
162.6–188.4
Run-level p95 TPOT range (ms)
15.7–15.7
Run-level p95 ITL range (ms)
17.1–17.7

RunPod / H100 SXM

Output tokens / sec across runs
3,709.5–3,732.8
Run-level p95 TTFT range (ms)
261.2–262.4
Run-level p95 TPOT range (ms)
8.0–8.1
Run-level p95 ITL range (ms)
8.6–9.4

Lambda Cloud / H100 SXM

Output tokens / sec across runs
3,793.7–3,796.4
Run-level p95 TTFT range (ms)
83.6–85.5
Run-level p95 TPOT range (ms)
8.2–8.2
Run-level p95 ITL range (ms)
8.4–8.4

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 15:40:19
Measurement finished (UTC)
2026-10-08 15:41:20
Measured repetition duration (seconds)
9.07–9.13
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 5e2781057f6315b8339a574b966ea8e944288b53367b3a47426074c78e87ad79
  • 91e233778bbec5085423d5c13cb700a1a648c32f2e72c160fdf8005e1d76ac30
  • a175ef7478ee1e7a922a3019670eec99bc0845a799af390bf696b868bc855015

Retained server-log SHA-256 identifier:

076ecae71a431a44e15e41eba444a44d456e911db17bde7137b48d9217de3fa1

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 08:53:59
Measurement finished (UTC)
2026-10-09 08:56:55
Measured repetition duration (seconds)
39.34–39.42
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 06d5d38945084f7ba6571dc065312df0093db15c9eac9309c64843ddf7df7b4b
  • d21893711f9c9250e380da1caefd52f161a1475e36705fec46af8c6113b44994
  • ebec1c61d306b96fe4a3ed7d634e9c130ed0166d87673e71752c141bdcf24743

Retained server-log SHA-256 identifier:

9365da7409284af289c728d481504a2f7214a4b86f345364999e932ef6f51228

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 10:17:32
Measurement finished (UTC)
2026-10-07 10:19:05
Measured repetition duration (seconds)
16.13–16.20
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 86350649fe6072b78357dc85240993b2076868943991717b5bda76e16ab8e840
  • aaf70baf4fae78a95581e52b62c717458c13daea5cfcdd504ec4c3cbb344b1bb
  • ddd36761d86d82ff80b271abe3012ef9824022a11f291c33b8fd1579e1b703d0

Retained server-log SHA-256 identifier:

6aaac55ca772f970ed22038083ba3cde4e9ffc03c3edb5055b95d00f4d6b19bb

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:30:47
Measurement finished (UTC)
2026-10-07 11:31:45
Measured repetition duration (seconds)
8.57–8.63
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 2b937286ea7f27c1fdf6684fd08819f027d147cbf8bd5a161f9bb2231998a39a
  • 9f1a53fcc0a1f8dcb3b35fdac2deed753700f9287f35da676ec38f866438f5f2
  • c7ed2bbb542fb24ec30f74d304b869d7354508643357ea65102b97a43425fa04

Retained server-log SHA-256 identifier:

48051e78f014738de60f410e08f3014a7e3e3f2c5cc9d45e39459eaa664b5c52

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:37:57
Measurement finished (UTC)
2026-10-08 13:38:51
Measured repetition duration (seconds)
8.43–8.44
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 50b6ddf6d5dc1cba2f3fe6ada9498c54ec057033edfbbd907e8e21c3695b8ffe
  • 7ad4cebb780c710a77497b8947645104ca28d019e3e8c5a8828677cafa278a98
  • b6579160d521dac46def0a5b5d49f39b810728959b5604fdd14b90867857cd4d

Retained server-log SHA-256 identifier:

7b8b99081e6adb3d6bca673ea4d142e9217f098b3ca26fe48aa4ed9c6a357a93

Long generation / decode-heavy — concurrency 8

512 input tokens, 2,048 output tokens, 8 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 64 completed requests per run (192 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 512 input and 2048 output tokens at concurrency 8
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $0.5760 1,138.1 91.5 7.0 7.4
RunPod / A100 SXM Standard protocol $0.6475 682.1 75.3 11.7 12.4
RunPod / A40 Standard protocol $0.6666 245.9 234.8 32.5 34.3
RunPod / H100 SXM Standard protocol $0.8157 1,188.6 70.4 6.8 7.4
Lambda Cloud / H100 SXM Standard protocol $1.0505 1,134.4 30.4 7.1 7.4

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $0.5760 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
1,138.1
Worst p95 TTFT (ms)
91.5
Worst p95 TPOT (ms)
7.0
Worst p95 ITL (ms)
7.4
p95 TTFT across runs (ms)
81.1–91.5
RunPod / A100 SXM — $0.6475 / 1M output
Evidence type
Standard protocol
Output tokens / sec
682.1
Worst p95 TTFT (ms)
75.3
Worst p95 TPOT (ms)
11.7
Worst p95 ITL (ms)
12.4
p95 TTFT across runs (ms)
68.7–75.3
RunPod / A40 — $0.6666 / 1M output
Evidence type
Standard protocol
Output tokens / sec
245.9
Worst p95 TTFT (ms)
234.8
Worst p95 TPOT (ms)
32.5
Worst p95 ITL (ms)
34.3
p95 TTFT across runs (ms)
135.7–234.8
RunPod / H100 SXM — $0.8157 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,188.6
Worst p95 TTFT (ms)
70.4
Worst p95 TPOT (ms)
6.8
Worst p95 ITL (ms)
7.4
p95 TTFT across runs (ms)
65.3–70.4
Lambda Cloud / H100 SXM — $1.0505 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,134.4
Worst p95 TTFT (ms)
30.4
Worst p95 TPOT (ms)
7.1
Worst p95 ITL (ms)
7.4
p95 TTFT across runs (ms)
27.3–30.4
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
1,137.1–1,138.2
Run-level p95 TTFT range (ms)
81.1–91.5
Run-level p95 TPOT range (ms)
7.0–7.0
Run-level p95 ITL range (ms)
7.4–7.4

RunPod / A100 SXM

Output tokens / sec across runs
682.1–682.1
Run-level p95 TTFT range (ms)
68.7–75.3
Run-level p95 TPOT range (ms)
11.7–11.7
Run-level p95 ITL range (ms)
12.4–12.4

RunPod / A40

Output tokens / sec across runs
245.8–245.9
Run-level p95 TTFT range (ms)
135.7–234.8
Run-level p95 TPOT range (ms)
32.5–32.5
Run-level p95 ITL range (ms)
34.2–34.3

RunPod / H100 SXM

Output tokens / sec across runs
1,177.5–1,204.2
Run-level p95 TTFT range (ms)
65.3–70.4
Run-level p95 TPOT range (ms)
6.6–6.8
Run-level p95 ITL range (ms)
7.3–7.4

Lambda Cloud / H100 SXM

Output tokens / sec across runs
1,133.3–1,134.4
Run-level p95 TTFT range (ms)
27.3–30.4
Run-level p95 TPOT range (ms)
7.0–7.1
Run-level p95 ITL range (ms)
7.4–7.4

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 15:52:54
Measurement finished (UTC)
2026-10-08 15:59:50
Measured repetition duration (seconds)
115.15–115.27
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 1454c53c9ec2c3f1da1316f07a8f730e63145bc96c8931aff4bd180a4d1731ff
  • 6a4571c571ab52a2fa5c5d8bbf742510f0c0ca686abf0f9c80dff344e96afcc9
  • daa2fed10931480cf2004077acbed138b216c67deca1732952606ba91c2c7131

Retained server-log SHA-256 identifier:

ae05580340216d2311a4fd668b0a37989b7713e99ec8e2943d8f3ff9646c23a5

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 10:56:36
Measurement finished (UTC)
2026-10-07 11:08:01
Measured repetition duration (seconds)
192.15–192.17
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 1d4619a66501f599008193bdc3c5915391635e1de87c8a467c1b2a1d98a91355
  • 8165f0ea83a139de9a0e92d5764d670da3fd9e87046d42767b89560c34dd9299
  • b9cc2aef9d3aed9fd059e217655dd0deddd4c4a7f21486865e13c05d9a3e221f

Retained server-log SHA-256 identifier:

88a0f03c6c0aa7dd07d3cabfa79b1964ce3575d45946828d9fc98457042475fa

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 11:40:42
Measurement finished (UTC)
2026-10-09 12:11:17
Measured repetition duration (seconds)
533.01–533.17
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 1ff6bb5c4083382611a6a49afb87f831477baa3bbcd16dfc20a10cfb4ad64e6f
  • 5ca88f00728a826c799fad57d8e078446b00d7aec18e3981df185b622d42289b
  • 9f5552d9cdd0f6e881984d89a6077caa7ae90950c092717c9c71d0baf9cb6561

Retained server-log SHA-256 identifier:

1ce583ff1f25c8fa1096e3e2a66585cf754dba840e3eb39d9810d38b2112dff4

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:56:18
Measurement finished (UTC)
2026-10-07 12:03:03
Measured repetition duration (seconds)
108.85–111.32
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 1a33cd4ba64be731835195b77daa069de0f7202c154e2912f32e48a0144cd4be
  • 4cde882cbcb37b572174f22cfa4449fb919dcd9a212a1daa09fad3b369a7ac70
  • 7d019111378946b05446d44d4d549444eb198b9d40a7ae4824fa0156234156a9

Retained server-log SHA-256 identifier:

0c84e2d20378631036591f92dd0e188a0e6cc7712c47087fc475a15de3641e40

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:50:11
Measurement finished (UTC)
2026-10-08 13:57:03
Measured repetition duration (seconds)
115.54–115.66
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 3819735efb5e8b2926190b67448a3ca49dfa31dda2739a02a1a2702a0bca591e
  • 6002f7308d84615714ebaee57d21afa7598d2ab76353e0d774146fd9171ea838
  • a2bce1ab4b3c90754474a887c72d6b762a2df7ba1f270ec91eaf6bf435217a02

Retained server-log SHA-256 identifier:

10d529d2c3f7143bb7478db723149e280ebeda2d81b3eb22c2b5b5d53de336e1

Long generation / decode-heavy — concurrency 32

512 input tokens, 2,048 output tokens, 32 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 128 completed requests per run (384 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 512 input and 2048 output tokens at concurrency 32
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $0.1814 3,613.0 213.8 8.8 10.1
RunPod / A40 Standard protocol $0.2173 754.1 2,186.6 42.3 47.9
RunPod / A100 SXM Standard protocol $0.2357 1,873.7 142.5 17.1 20.1
RunPod / H100 SXM Standard protocol $0.2543 3,811.8 109.2 8.4 10.2
Lambda Cloud / H100 SXM Standard protocol $0.3293 3,618.4 59.3 8.8 10.3

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $0.1814 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
3,613.0
Worst p95 TTFT (ms)
213.8
Worst p95 TPOT (ms)
8.8
Worst p95 ITL (ms)
10.1
p95 TTFT across runs (ms)
182.2–213.8
RunPod / A40 — $0.2173 / 1M output
Evidence type
Standard protocol
Output tokens / sec
754.1
Worst p95 TTFT (ms)
2,186.6
Worst p95 TPOT (ms)
42.3
Worst p95 ITL (ms)
47.9
p95 TTFT across runs (ms)
2,118.4–2,186.6
RunPod / A100 SXM — $0.2357 / 1M output
Evidence type
Standard protocol
Output tokens / sec
1,873.7
Worst p95 TTFT (ms)
142.5
Worst p95 TPOT (ms)
17.1
Worst p95 ITL (ms)
20.1
p95 TTFT across runs (ms)
115.4–142.5
RunPod / H100 SXM — $0.2543 / 1M output
Evidence type
Standard protocol
Output tokens / sec
3,811.8
Worst p95 TTFT (ms)
109.2
Worst p95 TPOT (ms)
8.4
Worst p95 ITL (ms)
10.2
p95 TTFT across runs (ms)
86.6–109.2
Lambda Cloud / H100 SXM — $0.3293 / 1M output
Evidence type
Standard protocol
Output tokens / sec
3,618.4
Worst p95 TTFT (ms)
59.3
Worst p95 TPOT (ms)
8.8
Worst p95 ITL (ms)
10.3
p95 TTFT across runs (ms)
58.1–59.3
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
3,607.7–3,613.0
Run-level p95 TTFT range (ms)
182.2–213.8
Run-level p95 TPOT range (ms)
8.8–8.8
Run-level p95 ITL range (ms)
10.1–10.1

RunPod / A40

Output tokens / sec across runs
753.3–754.7
Run-level p95 TTFT range (ms)
2,118.4–2,186.6
Run-level p95 TPOT range (ms)
42.2–42.3
Run-level p95 ITL range (ms)
47.8–47.9

RunPod / A100 SXM

Output tokens / sec across runs
1,872.2–1,875.0
Run-level p95 TTFT range (ms)
115.4–142.5
Run-level p95 TPOT range (ms)
17.0–17.1
Run-level p95 ITL range (ms)
20.0–20.1

RunPod / H100 SXM

Output tokens / sec across runs
3,809.9–3,813.6
Run-level p95 TTFT range (ms)
86.6–109.2
Run-level p95 TPOT range (ms)
8.4–8.4
Run-level p95 ITL range (ms)
9.9–10.2

Lambda Cloud / H100 SXM

Output tokens / sec across runs
3,616.7–3,618.4
Run-level p95 TTFT range (ms)
58.1–59.3
Run-level p95 TPOT range (ms)
8.8–8.8
Run-level p95 ITL range (ms)
10.3–10.3

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 16:01:27
Measurement finished (UTC)
2026-10-08 16:06:17
Measured repetition duration (seconds)
72.56–72.66
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 592cece443d47b072f6b4d3f177e07e3a262582441b1d35ef59af5aa6c3a1a16
  • 64595a19750c6b0586ddb74377494b14df9e893c116034ce814151388c931a47
  • a3dc6d0d00d0b5664cd6005680deb739f458cd6fa56ce7acbb2c31f1d0836111

Retained server-log SHA-256 identifier:

ae05580340216d2311a4fd668b0a37989b7713e99ec8e2943d8f3ff9646c23a5

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 12:18:18
Measurement finished (UTC)
2026-10-09 12:39:32
Measured repetition duration (seconds)
347.34–347.99
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 3c7bec73cfc3829058be7340508c8a5317670c14df8a69739f0b657f2ad5a85f
  • 3e8e3906e570e9e595dad392d4706cffcd7425a0142549d95887419cc459b271
  • ff810874a22fdb8d46de82014a92b5b77fe24963e0a6239cc861294fe51a500b

Retained server-log SHA-256 identifier:

1ce583ff1f25c8fa1096e3e2a66585cf754dba840e3eb39d9810d38b2112dff4

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:10:59
Measurement finished (UTC)
2026-10-07 11:19:47
Measured repetition duration (seconds)
139.81–140.02
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 3f82c52ea1b0c2afecc51a2f0745a8594ce4e13f2b25a798bd4658a768922b0b
  • d6511268c4aaa7930891fc6203a26ae95603cd952c875c69bb4c426763e8c9be
  • efc446f8bbd364d41eadbfea161db9ced59a05e9d0f0032c67211b2599b1841c

Retained server-log SHA-256 identifier:

88a0f03c6c0aa7dd07d3cabfa79b1964ce3575d45946828d9fc98457042475fa

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 12:04:37
Measurement finished (UTC)
2026-10-07 12:09:18
Measured repetition duration (seconds)
68.74–68.81
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 96c18bf0d52fbaa6836c418e46d7a32d3dfa0b0e26fafe32c6152b85ce2f2af9
  • 9736e51ffa47fd577daa6f4b6ae5e8e520e0dd6366d16e28bec2bb05d6c30756
  • cfa9c897a22ec274aa1652be0ea3d7a117af47ec845a6249fc066cf0537224ad

Retained server-log SHA-256 identifier:

0c84e2d20378631036591f92dd0e188a0e6cc7712c47087fc475a15de3641e40

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:58:39
Measurement finished (UTC)
2026-10-08 14:03:23
Measured repetition duration (seconds)
72.45–72.48
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 1d138a1218721144a98cca20e881064d05c88f7c60766c32cd37f6080519033b
  • b73f6055cc68a2db01ec3d30a8f6d4994c6497295adb76619f7c6830bfeef727
  • fde27873d054d8d769473264eed702ff2a0a990295ee5676b199c99df619a291

Retained server-log SHA-256 identifier:

10d529d2c3f7143bb7478db723149e280ebeda2d81b3eb22c2b5b5d53de336e1

Long-context / prefill-heavy — concurrency 8

8,192 input tokens, 256 output tokens, 8 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 64 completed requests per run (192 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 8192 input and 256 output tokens at concurrency 8
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $1.3126 499.4 1,747.2 14.8 9.9
RunPod / A40 Standard protocol $1.7640 92.9 7,171.4 79.1 363.0
RunPod / H100 SXM Standard protocol $1.9199 505.0 1,766.4 14.8 11.6
RunPod / A100 SXM Standard protocol $1.9518 226.3 2,710.0 32.2 164.2
Lambda Cloud / H100 SXM Standard protocol $2.3608 504.8 1,537.5 15.0 10.1

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $1.3126 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
499.4
Worst p95 TTFT (ms)
1,747.2
Worst p95 TPOT (ms)
14.8
Worst p95 ITL (ms)
9.9
p95 TTFT across runs (ms)
1,725.3–1,747.2
RunPod / A40 — $1.7640 / 1M output
Evidence type
Standard protocol
Output tokens / sec
92.9
Worst p95 TTFT (ms)
7,171.4
Worst p95 TPOT (ms)
79.1
Worst p95 ITL (ms)
363.0
p95 TTFT across runs (ms)
5,874.2–7,171.4
RunPod / H100 SXM — $1.9199 / 1M output
Evidence type
Standard protocol
Output tokens / sec
505.0
Worst p95 TTFT (ms)
1,766.4
Worst p95 TPOT (ms)
14.8
Worst p95 ITL (ms)
11.6
p95 TTFT across runs (ms)
1,752.1–1,766.4
RunPod / A100 SXM — $1.9518 / 1M output
Evidence type
Standard protocol
Output tokens / sec
226.3
Worst p95 TTFT (ms)
2,710.0
Worst p95 TPOT (ms)
32.2
Worst p95 ITL (ms)
164.2
p95 TTFT across runs (ms)
2,650.8–2,710.0
Lambda Cloud / H100 SXM — $2.3608 / 1M output
Evidence type
Standard protocol
Output tokens / sec
504.8
Worst p95 TTFT (ms)
1,537.5
Worst p95 TPOT (ms)
15.0
Worst p95 ITL (ms)
10.1
p95 TTFT across runs (ms)
1,297.7–1,537.5
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
499.1–500.0
Run-level p95 TTFT range (ms)
1,725.3–1,747.2
Run-level p95 TPOT range (ms)
14.2–14.8
Run-level p95 ITL range (ms)
9.8–9.9

RunPod / A40

Output tokens / sec across runs
92.9–92.9
Run-level p95 TTFT range (ms)
5,874.2–7,171.4
Run-level p95 TPOT range (ms)
79.0–79.1
Run-level p95 ITL range (ms)
362.6–363.0

RunPod / H100 SXM

Output tokens / sec across runs
503.8–505.6
Run-level p95 TTFT range (ms)
1,752.1–1,766.4
Run-level p95 TPOT range (ms)
14.8–14.8
Run-level p95 ITL range (ms)
10.8–11.6

RunPod / A100 SXM

Output tokens / sec across runs
225.7–226.3
Run-level p95 TTFT range (ms)
2,650.8–2,710.0
Run-level p95 TPOT range (ms)
32.1–32.2
Run-level p95 ITL range (ms)
162.6–164.2

Lambda Cloud / H100 SXM

Output tokens / sec across runs
504.5–504.9
Run-level p95 TTFT range (ms)
1,297.7–1,537.5
Run-level p95 TPOT range (ms)
14.9–15.0
Run-level p95 ITL range (ms)
10.0–10.1

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 15:43:08
Measurement finished (UTC)
2026-10-08 15:45:27
Measured repetition duration (seconds)
32.77–32.83
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 49926ec8f9f7877dcff9fa015c42476127168d0dd8f9e4edbbe1fd06d518351b
  • 58579ae8522f506976637eafbf0a2c31b86b899bf5329a523321183003ff2dd0
  • b9e43ffe3502c01f7d17bd300eca70f2b7a36e7113eb2e1b0cd4e6e5c113d659

Retained server-log SHA-256 identifier:

35852feb9c4f8ddce8570c8b691910f40d8971ca5e77da7b93c97eb801ac67df

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 09:07:50
Measurement finished (UTC)
2026-10-09 09:17:56
Measured repetition duration (seconds)
176.30–176.35
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 01c62089a7583844bc3ddf97a3e1e6f17df0f1f69ec56b623255931a537b5e6c
  • 0ac5227d90d7c9f92c8ae5eb4420ae3e239c9010e6cd53c86c137b1d2ab2ecbc
  • 672fbcfc6c66609a6af010a7beff9bf417f4bc8cf3a884801d69d8e98894d15f

Retained server-log SHA-256 identifier:

f6b5d65ed60d13f695bfa62a50e9cbbcbd8df6e3177e4d082c292a3accecde6d

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:39:50
Measurement finished (UTC)
2026-10-07 11:42:12
Measured repetition duration (seconds)
32.40–32.52
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 15794a83d5e549f4fe6ec5d23d76816056b7af3f6096a1658cc97ff434451ac8
  • 3d132b02b674edd331e8a8ae6bf4287062e0e36e2278d7840af56162c5097838
  • dbb8bd37ce9673cf778ca13a0bc4e88591366baf67ad8e951ac343960589fac6

Retained server-log SHA-256 identifier:

6d8e5ba4af426e23e787a9a0390e3bbb09a38e7e17cf80c60484c519e2150e5d

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 10:30:03
Measurement finished (UTC)
2026-10-07 10:34:38
Measured repetition duration (seconds)
72.40–72.59
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 9b03410bba4523e5e6b45eef51aa84f3e63b5a05bdaa6c6aededd7db360210ed
  • cf85669076d10b5cec638ba3cee423047eca74b9840cc11ffc228e9e28ae1cb6
  • e233d738133a46b35444a55fbbf11b996dfeb7a279b460c848b1dd79d3d4debd

Retained server-log SHA-256 identifier:

973c02e54015bfc2c61e0db4a815e387fd301ccd774793b26cbe386fa55977ca

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
64
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:40:40
Measurement finished (UTC)
2026-10-08 13:42:53
Measured repetition duration (seconds)
32.45–32.47
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 8a32721fad31bdd119bfdc31622345d2840774cfdbb6e19e8e200d9d87374176
  • cd616396de3f529cff9231d9b35360e71fe2d7b9b8f6ec06dfcd40c4ab46e41c
  • d0b141a013dc495b68bfea0612a6a8cf23b422bff32b6665845d9363260fbc4b

Retained server-log SHA-256 identifier:

e745fd520a996cbbaf07062fa88ce73d304bf76979e6b1301b1cbf9744ce0de6

Long-context / prefill-heavy — concurrency 32

8,192 input tokens, 256 output tokens, 32 concurrent requests. Five measured deployment classes.

Sample per deployment: 3 measured runs × 128 completed requests per run (384 requests in total). These are synthetic workloads; repeat variation is shown below.

If needed, scroll horizontally to view all metrics. Keyboard: focus the table region and use arrow keys.

Historical inference benchmark results for 8192 input and 256 output tokens at concurrency 32
Provider / GPU Evidence type Historical $ / 1M output Output tokens / sec Worst p95 TTFT (ms) Worst p95 TPOT (ms) Worst p95 ITL (ms)
Vast.ai / H100 SXM Reviewed historical $0.9018 726.9 5,692.4 40.9 219.9
RunPod / H100 SXM Standard protocol $1.3283 729.8 5,714.3 40.7 220.3
RunPod / A40 Standard protocol $1.5970 102.6 55,453.7 217.0 483.4
Lambda Cloud / H100 SXM Standard protocol $1.6373 727.8 5,649.0 41.6 222.5
RunPod / A100 SXM Standard protocol $1.7817 247.9 18,161.1 125.2 233.7

Expand a deployment to see its latency, throughput and evidence type.

Vast.ai / H100 SXM — $0.9018 / 1M output
Evidence type
Reviewed historical
Output tokens / sec
726.9
Worst p95 TTFT (ms)
5,692.4
Worst p95 TPOT (ms)
40.9
Worst p95 ITL (ms)
219.9
p95 TTFT across runs (ms)
5,677.4–5,692.4
RunPod / H100 SXM — $1.3283 / 1M output
Evidence type
Standard protocol
Output tokens / sec
729.8
Worst p95 TTFT (ms)
5,714.3
Worst p95 TPOT (ms)
40.7
Worst p95 ITL (ms)
220.3
p95 TTFT across runs (ms)
5,696.2–5,714.3
RunPod / A40 — $1.5970 / 1M output
Evidence type
Standard protocol
Output tokens / sec
102.6
Worst p95 TTFT (ms)
55,453.7
Worst p95 TPOT (ms)
217.0
Worst p95 ITL (ms)
483.4
p95 TTFT across runs (ms)
55,236.1–55,453.7
Lambda Cloud / H100 SXM — $1.6373 / 1M output
Evidence type
Standard protocol
Output tokens / sec
727.8
Worst p95 TTFT (ms)
5,649.0
Worst p95 TPOT (ms)
41.6
Worst p95 ITL (ms)
222.5
p95 TTFT across runs (ms)
5,606.7–5,649.0
RunPod / A100 SXM — $1.7817 / 1M output
Evidence type
Standard protocol
Output tokens / sec
247.9
Worst p95 TTFT (ms)
18,161.1
Worst p95 TPOT (ms)
125.2
Worst p95 ITL (ms)
233.7
p95 TTFT across runs (ms)
18,134.9–18,161.1
Variation between measured runs

Each range is the lowest to highest value observed across three runs. Figures are rounded to one decimal place; matching displayed endpoints can hide smaller differences between runs. Latency ranges compare the runs' p95 values; they are not ranges of individual request latencies, confidence intervals or production guarantees.

Vast.ai / H100 SXM

Output tokens / sec across runs
726.8–727.3
Run-level p95 TTFT range (ms)
5,677.4–5,692.4
Run-level p95 TPOT range (ms)
40.8–40.9
Run-level p95 ITL range (ms)
219.3–219.9

RunPod / H100 SXM

Output tokens / sec across runs
729.2–734.2
Run-level p95 TTFT range (ms)
5,696.2–5,714.3
Run-level p95 TPOT range (ms)
40.6–40.7
Run-level p95 ITL range (ms)
218.6–220.3

RunPod / A40

Output tokens / sec across runs
102.5–102.7
Run-level p95 TTFT range (ms)
55,236.1–55,453.7
Run-level p95 TPOT range (ms)
216.7–217.0
Run-level p95 ITL range (ms)
482.8–483.4

Lambda Cloud / H100 SXM

Output tokens / sec across runs
726.8–729.4
Run-level p95 TTFT range (ms)
5,606.7–5,649.0
Run-level p95 TPOT range (ms)
41.3–41.6
Run-level p95 ITL range (ms)
221.2–222.5

RunPod / A100 SXM

Output tokens / sec across runs
247.3–247.9
Run-level p95 TTFT range (ms)
18,134.9–18,161.1
Run-level p95 TPOT range (ms)
124.8–125.2
Run-level p95 ITL range (ms)
230.8–233.7

Small samples and short tests can miss rare slow requests and longer-term changes. How to interpret repeat variation.

Recorded serving configuration and historical rates

Vast.ai / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
2.3600
Evidence type
Reviewed historical
Protocol identifier
steady_state_v2_salvaged
Measurement started (UTC)
2026-10-08 15:46:28
Measurement finished (UTC)
2026-10-08 15:49:31
Measured repetition duration (seconds)
45.05–45.09
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 6597da5a06848d2c4584a5e09f22fe2b59efacaf07a8c315279fc7e25a36c891
  • 8023d89106153ea4604957ca0a8a53af40733d9fb2b628abf832981d10f1c217
  • cf258987c7b4afc48f516b63f17d3ce5633365e1fac1d4cb9ed357b823fa7c1b

Retained server-log SHA-256 identifier:

35852feb9c4f8ddce8570c8b691910f40d8971ca5e77da7b93c97eb801ac67df

RunPod / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
3.4900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 11:43:14
Measurement finished (UTC)
2026-10-07 11:46:20
Measured repetition duration (seconds)
44.63–44.94
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 5b16dce1e8880501caf9e4b5459a0b293b821b73db9a8777b9c45e52f3ca2d09
  • 610d0a737d4cdc8e82b5f9bd07cdb5677898a298d3fa9f7be5193005806af52f
  • ae229652535502716a4184cd5696c591ca935b567f8e8a86bc6b8fa6f3d42a64

Retained server-log SHA-256 identifier:

6d8e5ba4af426e23e787a9a0390e3bbb09a38e7e17cf80c60484c519e2150e5d

RunPod / A40

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
0.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-09 11:06:04
Measurement finished (UTC)
2026-10-09 11:23:26
Measured repetition duration (seconds)
319.03–319.61
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 117ae2d17d98bb6586f6cfe67da661bccabd84ba2a92a08d0087e1d48a21fcb0
  • 4bd88726c8602296ea2859e4d77e93f3e1edf7facb2d713c2784175110eecf3d
  • ad82c3f1380cca5769d1bf502784f068c25557da032e391189caea0d7925c7bf

Retained server-log SHA-256 identifier:

1d91e38788361d71ab9dc7e886311a02134b6efa0c3ecf82b272aa0a09fbcc9a

Lambda Cloud / H100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
4.2900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-08 13:43:52
Measurement finished (UTC)
2026-10-08 13:46:48
Measured repetition duration (seconds)
44.92–45.08
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 9e5cff4a02fcdee1547d5dbb3d92dea954edaedec55923a6bc0bcecf523b47ea
  • efebd1eff3235d54a4e142ebbd7fa82c78993fbc4b32a96125cf60676fb9add4
  • fd5e0b4517d47d2565dcd844fa072a68872475428987d67809755f59bcc04055

Retained server-log SHA-256 identifier:

e745fd520a996cbbaf07062fa88ce73d304bf76979e6b1301b1cbf9744ce0de6

RunPod / A100 SXM

Checkpoint
meta-llama/Llama-3.1-8B-Instruct
Precision
bfloat16
Serving engine
vLLM
Engine version
0.30.0
GPU count
1
Prompts per repetition
128
Historical hourly rate (USD)
1.5900
Evidence type
Standard protocol
Protocol identifier
steady_state_v2
Measurement started (UTC)
2026-10-07 10:37:12
Measurement finished (UTC)
2026-10-07 10:44:55
Measured repetition duration (seconds)
132.17–132.49
Dataset
random
Request rate
Unthrottled (inf)
Temperature
0
Benchmark warmups
8
Maximum sequence length (tokens)
131072
Tensor parallel size
1
Pipeline parallel size
1
Prefix caching
Enabled
Chunked prefill
Enabled
Quantization in engine log
None configured
Model revision in engine log
Not specified; immutable weight revision unverified
Tokenizer revision in engine log
Not specified; immutable tokenizer revision unverified

SHA-256 identifiers for the three original measured files:

  • 184e8c9035ba527708cf9ced3ef7cfc11aa779dac15569d27304b9ccf87971d9
  • 7786ca7023e7b14f3429372f0dd06f7466e38091e63146e9aaf30a4c95668a99
  • b6c05cc165839854a9c7b284c217b989c1cf3073c10ffdfdcf369c63d2732199

Retained server-log SHA-256 identifier:

973c02e54015bfc2c61e0db4a815e387fd301ccd774793b26cbe386fa55977ca

How to interpret the results

Latency matters: the RunPod A40 long-context workload at concurrency 32 recorded a worst p95 time to first token of 55,453.7 ms (about 55.45 seconds). Historical cost alone does not make that configuration suitable for interactive use.

Cost per million output tokens is calculated from the benchmark-time hourly price and measured output throughput. It excludes idle GPU time, provisioning, storage, networking, support, failed requests and other operational costs.

GPU marketplaces vary by host and region. Vast.ai measurements may reflect different hosts than the ones advertised at a later date. All measurements are tied to their recorded serving setup and are not performance guarantees for newly provisioned instances.

For a real deployment, the most economical eligible GPU must be selected against current prices, availability and the workload's actual latency requirements.

Read the full methodology · Register your interest