Measured output throughput
813.01 output tokens/second
Inference economics
How much does GPU inference cost per million tokens?
The cheapest hourly GPU is not necessarily the cheapest deployment for a given model, throughput requirement and latency target.
The output-token cost formula
Cost per million output tokens
(GPU hourly rate / 3600)
/ measured output tokens per second × 1,000,000
This estimates compute cost under the measured workload and benchmark-time hourly rate. It is not a total cost of ownership calculation.
A measured example: Llama 3.1 8B on A40
For 1,000 input tokens, 250 generated tokens and concurrency 32, our RunPod A40 benchmark recorded:
Historical compute cost
$0.2016 per million output tokens
Illustrative 1,000 requests
$0.0504 in generated-token compute cost at the measured throughput
This illustration assumes the measured serving configuration and 250 generated tokens per request. It excludes idle time and other operating costs. These are historical benchmark economics, not a current RunPod quotation.
The recorded historical hourly rate was $0.59/hour. Calculations use unrounded evidence values; displayed results are rounded.
Why cost per token can be misleading
Latency constraints
The minimum-cost GPU may exceed the application's p95 time-to-first-token or generation-latency requirements. A cheaper but infeasible deployment is not a valid recommendation.
Idle capacity
Benchmarked throughput is not equivalent to utilisation over an entire billing period. Real applications may pay for GPUs when no requests are being processed.
Input versus output tokens
Long prompts increase prefill work. Different input/output token shapes and concurrency levels can materially change performance and cost.
Infrastructure overhead
Storage, networking, failed requests, regional availability, deployment operations and downtime must be incorporated into all-in costs.
Choosing a cost-efficient deployment
Compare candidates using the same model, compatible serving configurations, actual request distributions and explicit SLO requirements. Then apply current provider price and availability observations before making a deployment decision.
Explore our inference benchmark catalog · Register your interest
Evidence checkpoint: c63d000. A benchmark does not guarantee performance on another host or at a later time.