On this page
A model request times out. Retrying once may succeed; retrying everywhere, immediately and indefinitely, can add load to a service that is already struggling. For an agent, the problem is broader than waiting between HTTP requests: a replay can consume more budget, produce a different answer, or repeat a tool’s side effects.
The following is a design guide, not a description of any particular provider’s guarantees. Check the API and client-library documentation for the actual error codes, limits, and idempotency behavior.
Decide whether another attempt is safe
Retryability has two independent questions: might the failure be transient, and is repeating this operation acceptable?
| Situation | Conservative starting point |
|---|---|
| Connection timeout or temporary service error | Consider a bounded retry if the operation is safe to repeat; the server may already have processed it |
| HTTP 429 rate limit | Respect provider guidance and any Retry-After; a quota or billing limit may require action rather than waiting |
| Invalid request, authentication or permission error | Do not repeat unchanged; surface the problem for correction |
| Partial streamed output | Do not silently append a fresh generation as though it were a continuation |
| A tool may already have changed state | Verify the outcome or use a supported idempotency mechanism before any replay |
An HTTP 429 response may include Retry-After; it does not have to. The header can contain a delay in seconds or an HTTP date. For a valid instruction from a service, wait at least the indicated interval; if the deadline does not permit that, fail or defer rather than retrying early. Use the service’s documented rules for interpreting its errors.
Even if an API request is safe to repeat in the HTTP sense, a model can return a different answer or charge for another attempt. A timeout does not prove that the first request was never processed. An agent that executes a returned tool call must track execution separately from model generation: retrying the generation is not permission to run an already-executed action twice.
Bound both attempts and time
Choose a total deadline and a maximum number of attempts, including the initial request. Apply per-attempt timeouts within the remaining deadline, and propagate cancellation. Also account for retries already performed by the SDK or gateway: three attempts in each of two independent layers can make up to nine downstream attempts for one user request.
For example, an interactive task might allow three total attempts, while a background job might have a different budget. These numbers are illustrations, not universal defaults. Limit concurrency and consider a shared retry budget so that many simultaneous requests cannot each amplify a provider outage.
Add jitter after the server’s minimum wait
For retry number , let the local jitter window be , with base and cap . Draw a delay uniformly from zero to . If the server asks for a minimum wait , one simple policy is to wait plus that random delay. The cap applies to the local jitter window, not to the server’s minimum.
This small, offline Python example calculates a delay; it does not send requests or parse HTTP headers. It expects a nonnegative retry number, a valid Retry-After already converted to seconds, and a caller-supplied random draw:
def retry_delay(n, remaining_s, retry_after_s, draw):
window = min(8.0, 0.5 * 2 ** min(n, 10))
minimum = max(0.0, retry_after_s or 0.0)
delay = minimum + draw(0.0, window)
return delay if delay < remaining_s else None
# A deterministic midpoint stands in for a random draw in tests.
midpoint = lambda low, high: (low + high) / 2
assert retry_delay(0, 5.0, 3.0, midpoint) == 3.25
assert retry_delay(10, 5.0, None, midpoint) == 4.0
assert retry_delay(0, 3.2, 3.0, midpoint) is None
In production, use a real random source and the client’s tested header parsing. Parse an HTTP-date relative to a suitable wall clock; use a monotonic clock for elapsed deadlines. After waiting, check the deadline and cancellation again and leave time for the next attempt. A retry that cannot fit should not start. Jitter reduces synchronization; it does not guarantee service recovery.
Separate per-request retries from a shared cooldown
A request retry handles one operation. A cooldown is a separate scheduling decision: after repeated qualifying failures, temporarily avoid sending new work to the affected resource, then allow limited probes when the cooldown ends. This resembles a circuit breaker, but a complete breaker also needs explicit states, concurrency control, and a recovery policy.
Scope matters. A timeout for one model does not prove every model is unhealthy; conversely, a rate limit may apply to a whole project or provider rather than one model. Choose the key and failure threshold from documented limits and observed evidence, not from the model name alone. Coordinate workers where needed, avoid synchronized probes, and test recovery as well as the transition into cooldown.
If another model is available, do not silently switch just because it is responding. Capabilities, privacy terms, cost, output format, and quality may differ. Fall back only under an explicit policy that permits that alternative, and tell the caller what happened.
A small test plan
Use a fake clock, deterministic random draws, and a stubbed provider; no real API traffic is needed to check these cases:
- A transient error followed by success stops retrying immediately after success.
- An invalid request is surfaced without an unchanged retry.
- A valid
Retry-Afterlonger than the remaining deadline causes a fail or deferral, never an early retry. - Cancellation stops waits and prevents further attempts; configured SDK retries do not exceed the overall budget.
- Concurrent failures do not trigger an unbounded flood; limited probes allow recovery without a synchronized surge.
- A partially delivered stream or an already-executed tool is not silently duplicated.
Measure attempt counts, delay, outcome, and cooldown transitions using appropriately redacted telemetry. Avoid recording prompts, outputs, or credentials by default. A retry policy should make failures easier to recover from and easier to understand when recovery is not possible.