How our inference benchmarks work

Workload identity

A benchmark comparison must match the model and served configuration, input token count, output token count, concurrency and relevant serving parameters. GPU class alone is not sufficient to establish comparable performance. The public Llama comparison uses matching input/output shapes and concurrency, with separately measured deployments.

How results qualify for publication

Every published result must have three successful measured repetitions and pass our internal evidence checks. The 30 results use two evidence types, identified in both the tables and mobile result panels:

• Standard protocol — 24 results. Measurements collected after an explicit stabilization phase using our standard benchmark procedure.
• Reviewed historical — six results. Earlier Vast.ai H100 measurements accepted after a separate audit of their procedure and source records. This review does not turn them into new measurements or mean they were rerun under the standard protocol.

These labels describe Exulux's internal evidence review, not independent third-party certification. They do not remove the differences in how the two evidence types were collected or the limitations described below.

Sample sizes and variation between runs

The comparison shows the completed-request count for each workload: three runs of 64 requests at concurrency 8, or three runs of 128 requests at concurrency 32. These are finite samples, not a long-duration reliability test.

The workloads use synthetic random-token prompts with controlled input and output lengths. They help compare the tested configurations, but do not reproduce a customer's prompts, arrival patterns, caching behaviour or mix of request lengths. They do not measure answer quality.

Each result's variation panel shows the minimum and maximum throughput and run-level p95 latency across the three repetitions. A p95 latency range compares three p95 values, not every individual request. It is not a confidence interval. The main table retains median throughput and the highest observed per-run p95 latency.

Three repetitions do not establish how reliably another deployment will meet a latency target. Small request samples, especially in short runs, may underrepresent rare slow responses. Validate production choices with representative traffic, longer tests and equivalent service requirements.

Latency

TTFT measures time to first token. TPOT measures time per output token. ITL measures inter-token latency. The comparison reports the highest observed p95 metric among successful measured repetitions. A historical p95 measurement is not a contractual SLO or a guarantee of tail latency for every deployment.

Cost methodology

Output-token compute cost uses observed benchmark throughput and the recorded hourly GPU rate. It does not represent the total operating cost of an application. Production utilisation, request mix, idle time, failed inference requests, storage and networking must be included in a customer-specific economic analysis.

Freshness and provider identity

Historical performance results and live GPU marketplace availability are different data sources. A marketplace quote can change or disappear without invalidating an older performance measurement. Performance on one Vast.ai host should not be represented as a guarantee for another host.

The cost rankings use prices recorded with the original measurements. They are not a current quote or availability feed. Applying a newly observed price to historical throughput would be a separately dated estimate, not a new performance measurement. This website does not currently provide that service.

Reproducibility and limitations

The comparison's expandable configuration sections disclose only allowlisted catalog fields and provenance verified against the 90 original measured files and their completed-run manifests. The measured runs span 7–9 October 2026 (UTC). Per-cell intervals include the gaps between repetitions; repetition duration is the benchmark's measured duration, not provisioning or stabilization time.

The configuration sections include dataset, request rate, temperature, warmups, measured duration ranges and SHA-256 file identifiers. Those hashes identify the original evidence; raw files containing generated text and operational metadata are not published. Retained engine initialization logs also establish maximum sequence length, tensor and pipeline parallelism, prefix caching, chunked prefill and configured quantization for each deployment. These are recorded benchmark settings, not recommendations from the private optimizer.

The logs report model and tokenizer revision as None: no revision was specified to the engine. This does not identify the immutable snapshot of the locally loaded weights or tokenizer. The summary does not establish immutable model revisions, regions or complete server startup settings, so this public summary alone is not sufficient to reproduce a deployment exactly. The promotion date below is not the measurement date. These missing details remain a publication limitation.

Throughput is the median across the accepted repetitions. Each latency column is the maximum of the per-repetition p95 values, not a pooled percentile across runs.

The comparison dataset is tied to the validated evidence checkpoint c63d000, promoted on 9 October 2026. The underlying measurements do not prove performance for all models, deployment regions, traffic patterns or serving software versions. A real customer evaluation should use representative request traces and equivalent reliability requirements.