IIntroduction to AI Systems Performance EngineeringTechnical textbook
Chapter 20: Production Inference Systems
20.4

Overload, failure, and graceful degradation

Keep overload bounded and recovery predictable

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

  • Admission control
  • Error budgets
  • Fault tolerance

One GPU replica fails while the service receives 110 requests per second. The remaining replicas can sustain only 85 useful completions per second under the latency objective. Without admission control, the queue grows by roughly 25 requests every second, first tokens arrive later, clients time out, and retries add more arrivals. A time-series plot of offered load, admitted load, queue length, completion rate, and retry rate shows the feedback loop.

An overload policy rejects, defers, degrades, or reroutes work before late requests consume key-value blocks and GPU time that could still complete other requests successfully. A failure drill can remove one replica, replay the same arrival stream, and record recovery time, rejection decisions, useful completions, and memory reserve. The normal case and the replica-loss case require different capacity margins.

Define service behavior for offered load above sustainable capacity. An unbounded queue causes increasing latency, memory pressure, timeouts, and retries. Client retries can add more load. Admission control rejects or defers work before the work consumes scarce resources.

Graceful degradation defines which service promises may change and how clients are informed. Lower-priority work may wait or route elsewhere. Maximum output or context may be bounded under a declared policy. A smaller or quantized model may serve only when its quality is approved for that class. Silent changes to model behavior are not an overload policy.

Failure and overload interact. A slow replica can cause client timeouts and retries, adding traffic to healthy replicas. A cache or network failure can leave partial generated output and external tool side effects. Recovery needs request identity, committed-progress semantics, and controlled retry behavior.

Bound admitted work before expensive execution

Use bounded queues and class-aware admission based on estimated prompt work, requested output, KV growth, deadline, and current reserve. Request count alone treats a 20-token prompt and a 100,000-token prompt as equal demand. Token or time budgets provide a closer estimate.

Reject or defer a request before prefill when predicted completion cannot meet its deadline or when its state would violate the free-block reserve. Continuing obsolete work wastes model and cache capacity. Cancellation should propagate to scheduler and GPU work as soon as safely possible.

Return explicit overload status, retry guidance, and randomized backoff where retry is appropriate. Apply retry budgets by request identity. If every client retries immediately after the same timeout, offered load increases precisely when capacity has fallen.

Protect fairness, isolation, and in-flight decode

Reserve capacity or scheduler share for requests that are already streaming when the product prioritizes completion consistency. New long prefills should not consume all token budget and KV blocks while active users stop receiving tokens. Decode protection needs a limit so a large active population does not starve every new request.

Use per-tenant quotas, weighted scheduling, maximum prompt and output limits, and separate pools when stronger isolation is required. Isolation controls prevent one tenant or request shape from consuming every cache block or scheduler iteration. The controls also define commercial and security boundaries.

Quality fallback must be explicit. Routing to a smaller model, reducing context, disabling tools, or changing precision can affect correctness and user expectations. Approve each fallback for stated request classes, surface it in telemetry or response metadata as policy requires, and count its quality outcome in goodput.

Define retry and committed-progress semantics

A request that has not produced output can often be retried more simply than one that has streamed tokens. Once output is visible, restarting generation can duplicate or contradict content. Tool calls, database writes, and other external actions need idempotency keys or application-level deduplication.

Track request identity, sequence identity, emitted tokens or bytes, and committed external actions. A transport reconnect can resume only when the server retains compatible state and the protocol defines the resume point. Otherwise return a clear failure rather than silently starting a second independent generation.

Use circuit breakers and health-based routing to stop sending work to a degraded dependency or replica. Drain unhealthy instances and preserve enough healthy capacity for recovery. Test sudden capacity loss with open-loop offered load and measure queue growth, rejection, retry amplification, goodput, and time to return to the normal operating region.

Recovery also requires memory cleanup. A failed worker can leave gateway state, scheduler reservations, prefix references, or remote KV transfers that no live request owns. Use leases, epochs, or another ownership protocol to prevent stale completion messages from reviving released state. Verify that repeated failure and restart does not leak cache capacity or duplicate completion.

Worked example: Stopping a retry storm

A replica slowdown causes client timeouts. Clients retry immediately, doubling offered load and pushing the entire pool into saturation.

  1. Set deadlines and reject work that cannot meet them before expensive prefill begins. Return explicit overload signals with randomized backoff guidance.
  2. Limit retries by request identity and ensure streamed or tool side effects are not duplicated. Route only to replicas with healthy queue and cache reserve.
  3. Reserve capacity for in-flight decode so new prefills cannot destroy TPOT for requests already being served.
  4. Run an overload test that removes capacity suddenly. Measure goodput, queue, rejection, retry amplification, and recovery time.

Conclusion: Early deadline-aware rejection, bounded retry, and reserved decode capacity prevent the slowdown from becoming a pool-wide retry storm. The overload test must show bounded queue size, controlled retry amplification, preserved goodput, and a predictable recovery interval.

Common errors

  • Using an unbounded queue as capacity. An unbounded queue stores delay and creates timeouts; the queue does not increase completed requests per second.
  • Retrying without idempotency. Generated streams and external actions can be duplicated or corrupted.
  • Degrading quality invisibly. Any model, context, or feature fallback belongs in an explicit product contract and telemetry.

Reference summary

Overload control uses bounded queues, deadlines, and class-aware admission based on estimated token work and future KV demand. Reject or defer work before expensive prefill when it cannot meet the service objective or would consume the reserve required by active requests.

Coordinate retry limits with admission control. Immediate retries can multiply offered load during a partial failure, consume the remaining reserve, and delay requests that would otherwise complete. Return explicit overload signals, add randomized delay, and cap retry attempts at the client or gateway.

  • Reject or defer before scarce KV allocation when predicted capacity is unavailable.
  • Use bounded queues and deadlines so obsolete work cannot accumulate indefinitely.
  • Cancel GPU work and free cache promptly when clients disconnect.
  • Route to smaller/quantized models only under an explicit quality policy.
  • Isolate tenants and priority classes to limit noisy-neighbor effects.
  • Canary engine, model, kernel, and driver changes with automatic rollback signals.
  • Preserve capacity for failover; a system that requires 100% of fleet for normal load has no recovery plan.

Knowledge check

How can an unbounded request queue cause timeouts and failed work in an interactive service?

Show the answer guide
  • When arrivals exceed service completions, an unbounded queue grows without limiting admission. Each new request begins behind more waiting work, so queue delay eventually consumes its latency deadline before execution starts.
  • If expired requests remain queued, the server can later prefill or decode work whose client has already timed out. Delayed cancellation wastes GPU time and reduces service available to live requests.
  • Queued requests and their metadata, tokenized prompts, reservations, or partial KV state can exhaust host or device memory. Longer queues also increase retry pressure when clients interpret delay as failure.
  • The feedback loop creates more timeouts and failed work even if raw GPU throughput remains high. Bounded admission, deadline-aware cancellation, backpressure, and prompt rejection preserve capacity for requests that can still complete.
Practice

Observe an overload queue and cancellation debt

Use a queueing simulator or test server with service capacity fixed near 100 requests per second. Send 150 requests per second for 120 seconds with a 2-second client deadline, then return to 20 requests per second. Compare an unbounded FIFO queue with a queue capped at 100 requests plus deadline-aware rejection and cancellation propagation.

Evidence to keep: Submit the simulator or load-test code and arrival trace, time-series plots of queue depth, live versus expired queued work, GPU or service utilization, timeout, rejection, goodput, and recovery time, plus an annotated request trace that receives service after its deadline in the unbounded case. Give a causal conclusion about wasted capacity and recovery delay.

The end-to-end optimization review is the method for deciding whether a kernel, scheduler, precision, topology, or reliability change improves this complete service.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.

  1. MLPerf Inference Rules↗
  2. PagedAttention / vLLM↗
  3. SGLang↗
  4. NCCL RAS↗
  5. NVIDIA Nsight Systems User Guide↗

    Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.