A40 vs H100 for Llama 3.1 8B: cost and latency

What we compared

Both deployments served Llama 3.1 8B Instruct in bfloat16 using vLLM 0.30.0, with one GPU. Three input/output token shapes were tested at concurrency 8 and 32. Concurrency means the configured maximum simultaneous requests in these finite runs, not a count of production users.

Each result contains three measured runs using synthetic random-token prompts. Each run completed 64 requests at concurrency 8 (192 total per deployment and workload), or 128 at concurrency 32 (384 total). The recorded benchmark-time rates were $0.59/hour for A40 and $3.49/hour for H100 SXM.

Hardware, host and serving conditions can all affect performance. Measurements were collected on different dates within the stated window; this is not an isolated test of GPU architecture.

All six matched cost and latency comparisons

Costs are historical USD per million output tokens, calculated from the recorded hourly rate and median aggregate output throughput. First-token latency is the highest run-level p95 across three runs, in seconds; lower is better. Calculations use unrounded values; displayed figures are rounded.

On narrower desktop layouts, scroll the comparison horizontally.

All six matched workloads are shown. They are selected grid points, not estimates of how often either GPU will be cheaper in real applications. Inspect all five deployments and repeat ranges.

Why the higher hourly rate sometimes paid off

For 1,000 input tokens, 250 output tokens and concurrency 32, A40 cost $0.2016 per million output tokens versus $0.2609 for H100 SXM.

For 8,192 input tokens, 256 output tokens and concurrency 32, the ranking reversed: $1.5970 for A40 versus $1.3283 for H100 SXM. H100's historical token cost was 16.8% lower.

Median aggregate output throughput was 102.63 tokens/second on A40 and 729.83 on H100 SXM. H100's hourly rate was 5.92 times A40's, while throughput was 7.11 times as high.

This allocates measured compute time to generated tokens, including the effect of input processing on achieved throughput. It excludes idle time and other operating costs. How to interpret cost per million tokens.

First-token latency changes the decision

H100 SXM had lower worst-run p95 first-token latency in every matched comparison. For long prompts at concurrency 32, A40 recorded 55.454 seconds versus 5.714 for H100 SXM.

The three run-level p95 values ranged from 55.236–55.454 seconds on A40 and 5.696–5.714 seconds on H100. These are ranges of three p95 statistics, not confidence intervals or the range of individual request latencies.

Time to first token measures the wait before generation begins. It does not describe response completion time or an individual user's generation speed. The complete results include additional latency metrics.

What this means for choosing a deployment

A40's lower historical token cost may make it a candidate for workloads resembling our short-generation tests if latency is acceptable. H100 combined lower cost and lower first-token latency in our long-prompt test at concurrency 32. At concurrency 8 for the same token shape, A40 retained lower cost while H100 had lower latency.

Start with acceptable first-token and generation latency, then compare costs among deployments that meet those requirements. Re-test with representative traffic and check current prices and availability before committing to infrastructure. Neither deployment is established as the universal best choice.

Limits of these results

This comparison covers one model, two RunPod GPU configurations and six synthetic settings. It does not establish results for other models, providers, quantisation settings or traffic distributions. Three finite runs with 64 or 128 requests each provide limited evidence for production tail latency.

Customer answer quality, long-term reliability, cold starts, idle utilisation and full operating costs are not measured. Immutable model/tokenizer revisions, regions and complete server startup settings remain unverified. Repeat variation does not cover those unmeasured differences.

Hourly rates belong to the original measurements. Replacing them with today's prices would create a separately dated estimate, not a new performance benchmark. Read the methodology and provenance.

Inspect the evidence or suggest the next comparison

Full Llama 3.1 8B results · Email a research suggestion.

Exulux currently provides historical benchmark research and email enquiries. Public optimisation-tool access is not yet available.