Explainer

Retrying Model API Calls Without Making an Outage Worse

A practical guide to retry budgets, jitter, Retry-After, scoped cooldowns, and the limits of replaying model requests and tool calls.

On this page

A model request times out. Retrying once may succeed; retrying everywhere, immediately and indefinitely, can add load to a service that is already struggling. For an agent, the problem is broader than waiting between HTTP requests: a replay can consume more budget, produce a different answer, or repeat a tool’s side effects.

The following is a design guide, not a description of any particular provider’s guarantees. Check the API and client-library documentation for the actual error codes, limits, and idempotency behavior.

Decide whether another attempt is safe

Retryability has two independent questions: might the failure be transient, and is repeating this operation acceptable?

SituationConservative starting point
Connection timeout or temporary service errorConsider a bounded retry if the operation is safe to repeat; the server may already have processed it
HTTP 429 rate limitRespect provider guidance and any Retry-After; a quota or billing limit may require action rather than waiting
Invalid request, authentication or permission errorDo not repeat unchanged; surface the problem for correction
Partial streamed outputDo not silently append a fresh generation as though it were a continuation
A tool may already have changed stateVerify the outcome or use a supported idempotency mechanism before any replay

An HTTP 429 response may include Retry-After; it does not have to. The header can contain a delay in seconds or an HTTP date. For a valid instruction from a service, wait at least the indicated interval; if the deadline does not permit that, fail or defer rather than retrying early. Use the service’s documented rules for interpreting its errors.

Even if an API request is safe to repeat in the HTTP sense, a model can return a different answer or charge for another attempt. A timeout does not prove that the first request was never processed. An agent that executes a returned tool call must track execution separately from model generation: retrying the generation is not permission to run an already-executed action twice.

Bound both attempts and time

Choose a total deadline and a maximum number of attempts, including the initial request. Apply per-attempt timeouts within the remaining deadline, and propagate cancellation. Also account for retries already performed by the SDK or gateway: three attempts in each of two independent layers can make up to nine downstream attempts for one user request.

For example, an interactive task might allow three total attempts, while a background job might have a different budget. These numbers are illustrations, not universal defaults. Limit concurrency and consider a shared retry budget so that many simultaneous requests cannot each amplify a provider outage.

Add jitter after the server’s minimum wait

For retry number n=0,1,2,…n=0,1,2,\ldots, let the local jitter window be wn=min⁡(c,b2n)w_n=\min(c, b2^n), with base bb and cap cc. Draw a delay uniformly from zero to wnw_n. If the server asks for a minimum wait rr, one simple policy is to wait rr plus that random delay. The cap applies to the local jitter window, not to the server’s minimum.

This small, offline Python example calculates a delay; it does not send requests or parse HTTP headers. It expects a nonnegative retry number, a valid Retry-After already converted to seconds, and a caller-supplied random draw:

def retry_delay(n, remaining_s, retry_after_s, draw):
    window = min(8.0, 0.5 * 2 ** min(n, 10))
    minimum = max(0.0, retry_after_s or 0.0)
    delay = minimum + draw(0.0, window)
    return delay if delay < remaining_s else None


# A deterministic midpoint stands in for a random draw in tests.
midpoint = lambda low, high: (low + high) / 2
assert retry_delay(0, 5.0, 3.0, midpoint) == 3.25
assert retry_delay(10, 5.0, None, midpoint) == 4.0
assert retry_delay(0, 3.2, 3.0, midpoint) is None

In production, use a real random source and the client’s tested header parsing. Parse an HTTP-date relative to a suitable wall clock; use a monotonic clock for elapsed deadlines. After waiting, check the deadline and cancellation again and leave time for the next attempt. A retry that cannot fit should not start. Jitter reduces synchronization; it does not guarantee service recovery.

Separate per-request retries from a shared cooldown

A request retry handles one operation. A cooldown is a separate scheduling decision: after repeated qualifying failures, temporarily avoid sending new work to the affected resource, then allow limited probes when the cooldown ends. This resembles a circuit breaker, but a complete breaker also needs explicit states, concurrency control, and a recovery policy.

Scope matters. A timeout for one model does not prove every model is unhealthy; conversely, a rate limit may apply to a whole project or provider rather than one model. Choose the key and failure threshold from documented limits and observed evidence, not from the model name alone. Coordinate workers where needed, avoid synchronized probes, and test recovery as well as the transition into cooldown.

If another model is available, do not silently switch just because it is responding. Capabilities, privacy terms, cost, output format, and quality may differ. Fall back only under an explicit policy that permits that alternative, and tell the caller what happened.

A small test plan

Use a fake clock, deterministic random draws, and a stubbed provider; no real API traffic is needed to check these cases:

  1. A transient error followed by success stops retrying immediately after success.
  2. An invalid request is surfaced without an unchanged retry.
  3. A valid Retry-After longer than the remaining deadline causes a fail or deferral, never an early retry.
  4. Cancellation stops waits and prevents further attempts; configured SDK retries do not exceed the overall budget.
  5. Concurrent failures do not trigger an unbounded flood; limited probes allow recovery without a synchronized surge.
  6. A partially delivered stream or an already-executed tool is not silently duplicated.

Measure attempt counts, delay, outcome, and cooldown transitions using appropriately redacted telemetry. Avoid recording prompts, outputs, or credentials by default. A retry policy should make failures easier to recover from and easier to understand when recovery is not possible.

Sources