Capacity and unit economics

Convert demand into active work and state

For each request class, begin with its arrival process, prompt-length distribution, generated-length distribution, latency objective, cache-hit behavior, and cancellation. Convert arrivals into input tokens per second, output tokens per second, active decode-sequence time, peak concurrent KV tokens, and network traffic.

In a stable window, Little's law provides a useful check: mean requests in the system equal arrival rate multiplied by mean residence time. Apply the relation separately to relevant classes or phases when their service times differ. Little's law predicts mean concurrency, not the KV-cache tail; memory admission still needs the joint distribution of concurrent sequence lengths and a reserve for growth.

Use measured operating curves for the selected model, hardware, parallel plan, quantization, and scheduler. A replica's sustainable goodput is the highest rate that meets the class SLO under the representative mix and remains stable. Offline maximum throughput is not the service rate for an interactive pool.

Traffic distribution creates fragmentation. Models, adapters, regions, tenants, and latency classes may need separate replicas or routing rules. Ten half-used specialized pools do not combine into five fully used replicas automatically. Record stranded memory and compute as part of fleet utilization.

Add reserve, burst, startup, and failure

Choose a utilization target that leaves room for normal length variation and arrival bursts. Then add the failure policy: loss of one replica, a percentage of a rack, or a complete zone. Divide required service rate by the capacity that remains in the stated failure case, and account for load-balancing unevenness.

Autoscaling cannot remove the need for immediate reserve. New replicas may need image download, model loading, quantization artifact loading, graph compilation, memory profiling, graph capture, health checks, and cache warming. If this takes minutes, existing reserve or bounded queues must absorb demand during that interval.

Stateful warmth affects restored capacity. A newly started replica has no prefix cache and may compile new shapes under load. A failover can therefore increase prefill demand at the same moment that fewer replicas remain. Test the failure transition, not only the final healthy replica count.

Calculate fully loaded cost per useful output

Include provisioned accelerator, host, memory, storage, network, energy, software, idle reserve, failures, and retries over a defined period. Divide by requests or tokens that meet quality and SLO requirements. Report marginal cost separately when a decision concerns one additional request on already provisioned capacity.

For purchased hardware, specify depreciation period, expected lifetime utilization, power, cooling, and maintenance. For cloud capacity, specify reservation terms, on-demand premiums, data transfer, and minimum billing intervals. Do not compare one option's marginal cost with another option's fully loaded cost.

Engineering complexity is difficult to reduce to one currency value but should remain explicit. A specialized topology can reduce accelerator expense while increasing rollout risk, on-call load, or vendor dependence. Present those risks beside the quantitative cost rather than hiding them in an arbitrary score.

Calculate cost over the complete demand curve. A configuration with the lowest cost per token at full load may be expensive during nights or regional troughs because its replicas cannot consolidate, scale down, or share capacity. Weight each operating region by time and traffic, then include the reserve that remains provisioned but idle.