AI inference benchmarks matter only when they show the operating tradeoffs a production service must make. A single tokens-per-second result can reward huge batches that no interactive user will tolerate. A latency-only score can reward an expensive, under-used cluster. InferenceMAX is more useful because it maps the curve between throughput, interactivity, cost, and power across real serving configurations.
That is the right question for an infrastructure team. Not which GPU wins one headline number, but which hardware and software combination meets a specific p95 target at a known cost per completed task. The answer changes with model architecture, context length, output length, concurrency, quantization, framework release, and network topology.
AI inference benchmarks need a Pareto curve
A Pareto frontier marks the configurations where improving one outcome requires sacrificing another. In LLM serving, the common tradeoff is aggregate throughput against per-user decode speed or time to first token. Batching more requests usually lifts accelerator utilization. It can also extend queue time and make a chat interface feel stalled.
Consider two configurations of the same 70B model. Configuration A delivers 15,000 output tokens per second but users see 20 tokens per second after a long queue. Configuration B delivers 9,000 output tokens per second with 55 tokens per second at the user. Neither figure wins by itself. A code assistant, agent loop, or support chat product may choose B. An offline document pipeline may choose A. The curve makes that choice visible.
This also stops a familiar procurement error: comparing a GPU peak number with a deployed endpoint result. Peak math rate says little about KV-cache pressure, expert routing, host scheduling, NIC contention, or the tail behavior created by mixed prompt lengths. A useful benchmark publishes the serving recipe and enough points on the curve to show where it bends.
What InferenceMAX adds to the benchmark stack
InferenceMAX is an open-source automated benchmark designed around LLM inference rather than a fixed chip specification. Its initial v1 coverage includes GB200 NVL72, B200, MI355X, H200, MI325X, H100, and MI300X systems. It tests multiple serving stacks, including SGLang, TensorRT-LLM, and vLLM, and republishes benchmark sweeps frequently. That cadence matters because kernels, schedulers, and parallelism strategies can move performance between hardware launches.
The published workload matrix includes DeepSeek-R1, gpt-oss-120b, and Llama 3.3 70B. It uses 1K/1K chat, 8K/1K summarization, and 1K/8K deep-reasoning input/output shapes. Those shapes expose different bottlenecks. A short chat turn leans toward responsiveness. An 8K input makes prompt processing and memory traffic more important. A 1K/8K generation makes decode efficiency and cache residency dominate.
Traditional suites still matter. MLPerf Inference Datacenter defines load patterns, latency constraints, compliance rules, and standardized result reporting. It answers whether a system processes a defined workload under a comparable protocol. InferenceMAX complements it by plotting more of the serving space and by tracking software revisions. Teams should use both: a controlled suite for apples-to-apples validation, then a Pareto sweep for a deployment decision.
The numbers need context, not blind trust
The early published results show why the curve is worth inspecting. NVIDIA reports that GB200 NVL72 reached a 15x advantage over H200 for a DeepSeek-R1 8K/1K configuration in its InferenceMAX analysis. The same report says a B200 configuration reached 10,000 tokens per second at 50 tokens per second per user on Llama 3.3 70B, more than four times H200 per-GPU throughput in that setup. These are vendor-reported figures, not a universal ranking.
The more instructive result is change over time. NVIDIA describes gpt-oss-120b cost per million tokens dropping from $0.11 to $0.02 at 100 tokens per second per user in two months through TensorRT-LLM changes. It also reports speculative decoding tripling throughput in one configuration. If a buying decision rests on a benchmark run from last quarter, that decision may be measuring old software as much as hardware.
The hardware details explain part of the gain. Blackwell systems pair fifth-generation Tensor Cores and low-precision formats such as NVFP4 with HBM3e and 1,800 GB/s bidirectional NVLink bandwidth in the cited configuration. For mixture-of-experts models, fast GPU-to-GPU movement matters because expert tokens must be routed and reassembled. Still, low precision, parallelism, and routing policies can change quality, latency, and operational complexity. A fast point that fails an accuracy check or breaches a p95 service-level objective is not a useful point.
Turn the chart into a capacity decision
Start with production traces, not a synthetic average prompt. Bucket requests by input tokens, expected output tokens, model, tool-call frequency, and priority. Include cold-cache and warm-cache runs. Include burst arrival patterns. An agent workload that alternates 500-token prompts with 4,000-token tool summaries will not behave like a steady stream of 1K/1K chat turns.
Set the constraints before measuring
- Define p50, p95, and p99 time to first token and decode-rate targets.
- Measure completed tasks and quality checks, not tokens alone.
- Record queue time, rejected requests, cache hit rate, and retry rate.
- Fix model revision, quantization, framework version, driver, and parallelism layout for each run.
- Measure wall power and cluster utilization when energy or rack capacity limits the deployment.
Then sweep concurrency and batch limits. Plot cost per successful task against p95 latency, with throughput and watts as labels. Keep every run’s configuration beside the plot. A point that looks cheap may depend on a 64-request concurrency level that the application rarely reaches. A point that looks responsive may leave expensive accelerators idle during the median hour.
Run the same procedure across candidate frameworks. TensorRT-LLM, vLLM, and SGLang differ in scheduling behavior, cache management, and support for new model features. Framework choice can move the frontier without changing the GPU. For multi-node or MoE deployments, repeat the sweep after changing tensor, pipeline, data, and expert parallelism. Network fabric can shift the result more than a small clock-rate difference.
What developers should take from InferenceMAX
Developers need a benchmark that predicts product behavior. That means testing a real request mix, preserving quality gates, and reporting user-visible latency. Infrastructure teams need the same evidence to set autoscaling triggers, select rack topology, and estimate capacity. The useful result is a set of defensible operating points, not a trophy score.
InferenceMAX is valuable because it treats serving as a moving systems problem. Its nightly-style approach recognizes that a kernel update, quantization path, or scheduler patch can matter as much as a new accelerator. Read the curve, reproduce the recipe, and choose the point that matches the service. Run and scale models on Wiro.
1 Comment