AI API error handling: timeouts, retries and duplicate work

Handle AI API failures by classifying the outcome before retrying. Invalid inputs, invalid credentials and exhausted funds need a correction. Temporary overload may justify a bounded retry. A disconnected or timed-out request has an uncertain outcome: inference may have completed after the client stopped waiting.

This distinction matters because generation consumes resources and may be billed. A retry policy is part of your cost and reliability design, not merely a convenience wrapper.

Separate four failure boundaries

First, a local validation failure means no request was sent. Second, a connection failure may occur before or after the server receives the request; a client often cannot prove which. Third, an HTTP failure provides a status but still needs interpretation. Fourth, a stream can start successfully and fail after partial output.

Keep these boundaries visible in application state. For example, use not_sent, failed, partial and uncertain states rather than reducing everything to an empty string. A UI that silently replaces a partial answer with a second generation hides what the user has already seen.

Use status-specific recovery

Status or condition	Suggested recovery	Retry automatically?
400 invalid request	Fix schema, model or parameters	No
401 authentication	Replace or configure the credential	No
403 permission	Check key model and modality scope	No
402 insufficient quota	Inspect balance and spend caps	No
429 rate limit	Reduce pressure and respect a valid retry hint	Only within a budget
503 unavailable	Check service status and bounded recovery	Possibly
Connection timeout	Preserve uncertain outcome	Only after duplicate-work review
Partial SSE output	Preserve partial text; show interruption	Never silently restart

These are recovery recommendations, not a promise that every deployment emits identical headers or error text. Consult the AIx error reference for documented envelopes. Do not branch on a provider’s prose message when an HTTP status or stable error type is available.

Bound the complete retry budget

Specify a maximum attempt count, a total elapsed-time budget and a concurrency limit. Exponential backoff with jitter spreads load, but it does not make an otherwise unsafe retry safe. Stop retrying when the user cancels, the deadline expires or the remaining budget is smaller than another plausible attempt.

A valid Retry-After may be a delay in seconds or an HTTP date. Parse it conservatively. Do not assume the header always exists, and do not wait indefinitely because a malformed value was accepted. Record attempt counts without copying the entire request into analytics.

For a first integration, the Python client makes one attempt and surfaces failures explicitly. This conservative default is easier to reason about than an invisible retry loop. Add retry behavior only after tests cover your chosen failure boundaries.

Idempotency is a contract with limits

The inspected AIx chat implementation reads Idempotency-Key. For completed non-streaming requests, a configured Redis cache can replay a stored response for up to 24 hours. Streaming uses a duplicate guard rather than replaying an entire stream.

These mechanisms depend on infrastructure configuration. They do not establish a universal exactly-once guarantee, and a response cache may not close the race between simultaneous first requests. Do not promise your customers that any arbitrary retry can never be billed twice.

Use one key for one logical operation and retain the original payload. Never reuse the key for a different prompt. Because replay caching can retain completion content, review data handling before processing sensitive material.

Test the states your users will see

Create fixtures for a successful response, invalid JSON, an empty choices array, 401, 402, 429, 503, a network exception and a partial stream. Assert that secret headers never appear in logs. Assert the attempt count and cancellation behavior rather than merely checking that some final text was returned.

For a permitted staging integration, test what happens when the upstream stalls and the client disconnects. Compare usage records with client events; do not infer billing from the browser alone. Mock tests establish application behavior against fixtures, while staging tests establish behavior for the tested deployment and date.

Operational checklist

Publish a user-visible retry action for uncertain results, retain a sanitized correlation identifier when available, and keep a recovery path for partial output. Monitor aggregate failure rates by status without collecting private prompts. Review retries when you change gateways, models or SDK versions; compatibility at the request-shape level does not guarantee identical recovery semantics.