Reliable AI chatbots do not assume that every model request will succeed. This runbook shows how to estimate timeout budgets, choose retry behavior, handle rate limits, route to a fallback model, recover from interrupted streams, and record enough context to diagnose failures. Use the formulas, decision tree, worked examples, and checklist before launch and during incident reviews.
Overview
AI chatbot error handling is a control problem, not just an exception-handling problem. A useful recovery strategy must protect four things at once: the user experience, the application's latency budget, the provider's limits, and the integrity of the operation.
When a request fails, first classify the failure. A temporary network interruption, a provider timeout, and a rate-limit response may justify a carefully bounded retry. Invalid authentication, an unsupported model, a malformed request, or a policy refusal generally requires a different path. Retrying every error increases latency and can turn a small incident into a larger one.
A practical decision tree is:
- Did the request reach the provider? If not, check connection, DNS, TLS, and local network errors.
- Is the error likely temporary? Consider timeouts, connection resets, service-unavailable responses, and rate limits. Retry only when the operation is safe and the retry budget remains.
- Can the request use a fallback? Route to a pre-tested model or provider only when it supports the required context, tools, output format, and safety controls.
- Is the response unusable? Validate the response before displaying or acting on it. A successful HTTP response is not necessarily a successful chatbot response.
- Can the user continue safely? Preserve the conversation where possible, explain what happened without exposing internal details, and offer a clear next step.
Keep recovery logic separate from prompt logic. A system prompt can guide answer quality, but it cannot replace request timeouts, idempotency controls, circuit breakers, validation, or observability.
How to estimate
Start with a user-facing latency budget. Let B be the maximum time your application can spend on the complete assistant turn. Reserve time for authentication, retrieval, tool calls, validation, rendering, and network overhead before assigning time to the model request.
A simple budget is:
model_budget = total_budget - application_overhead - tool_budget - safety_margin
Do not treat this as a promise that the provider will respond within the number you choose. It is an upper bound enforced by your application. If a request reaches that limit, cancel it and move to a defined recovery path.
Retry budget
For a request with a per-attempt timeout of T, retry delays of D1 through Dn, and n attempts, estimate worst-case waiting time as:
worst_case = (n × T) + D1 + D2 + ... + Dn
Use this calculation before enabling retries. If the result exceeds the user-facing budget, reduce the number of attempts, shorten the per-attempt timeout, or make the fallback path available earlier. Add jitter to backoff delays so many clients do not retry simultaneously.
A common backoff model is:
delay = min(max_delay, base_delay × 2^attempt) + random_jitter
The exact values should come from your service objectives and provider guidance rather than a universal recipe. Honor a provider's retry-after signal when it is available, subject to your own maximum delay and total budget.
Expected completion and capacity
Retries consume capacity. For an initial request with an estimated retry probability p, a rough expected-attempt estimate is:
expected_attempts ≈ 1 + p + p²
This simplified expression assumes at most two retries and independent failures, so it is only a planning estimate. Use production logs to replace it with observed rates. Track attempts per user turn, not only requests per minute, because one conversation turn can create several provider calls.
Inputs and assumptions
Document these inputs in configuration or an operational runbook. Avoid scattering them through application code.
- Operation type: Classify the request as conversational, retrieval, tool execution, batch processing, or another workflow. A tool call that changes data needs stricter retry rules than a read-only answer.
- Total latency budget: Set the maximum acceptable time for the complete user turn.
- Per-attempt timeout: Include connection, response, and streaming behavior. A streaming request needs an idle-read timeout as well as a total timeout.
- Retryable conditions: Define them by error class and request state, not by a broad rule such as “retry all five-hundred errors.”
- Maximum attempts and delay: Set both a count and a wall-clock limit.
- Idempotency strategy: Use an idempotency key or operation identifier for actions that might be repeated after an ambiguous timeout. Before retrying, determine whether the first request may already have succeeded.
- Fallback capability: Record which models or providers support the required context length, tools, structured output, language, and safety behavior.
- Response validation: Check for empty content, truncated output, invalid JSON, missing required fields, unsupported tool calls, and refusals that your workflow cannot process.
- Observability: Log a correlation ID, conversation or request ID, provider, model, attempt number, error category, latency, token or usage data when available, fallback decision, and final outcome. Redact prompts and personal data according to your data-handling rules.
Use a circuit breaker when repeated provider failures make retries wasteful. A basic breaker has three states: closed while requests flow normally, open while calls are blocked for a cooling period, and half-open while a limited probe tests recovery. The breaker should be scoped carefully; an outage for one model or region should not automatically disable unrelated routes.
Worked examples
Example 1: A transient timeout
Suppose a chatbot turn has a total budget of 12 seconds. Application work is estimated at 2 seconds, retrieval at 2 seconds, and the safety margin at 1 second. The model budget is therefore:
12 - 2 - 2 - 1 = 7 seconds
A design with two attempts, each limited to 3 seconds, plus a 1-second backoff has a rough maximum of 7 seconds. It leaves no room for unexpected overhead, so the safer design is to shorten the first attempt, remove the retry for interactive traffic, or invoke a fast fallback before the full budget is consumed. The calculation exposes the trade-off before users experience it.
Example 2: A rate limit
If the provider reports a rate limit, do not immediately send several parallel retries. Read the retry guidance when supplied, add jitter, and check whether the remaining user budget allows another attempt. If not, return a temporary, actionable message and place non-urgent work on a queue. For interactive traffic, a fallback should be selected only if its capacity and behavior are known.
Example 3: An interrupted stream
When streaming stops after partial output, mark the response as incomplete. Do not silently append a second model response to the partial text. You can retry from the original request, ask the model for a continuation with clear state, or show the partial answer with a retry control. The correct choice depends on whether duplication would be harmful and whether the response triggers tools or downstream actions.
Example 4: Invalid structured output
If the provider returns valid transport data but the content fails schema validation, classify it as an application-level failure. Record the validation error, then use a bounded repair attempt only if the task is safe and the remaining budget permits it. For high-impact workflows, stop and request human review rather than allowing malformed data to trigger an action. See Structured JSON Outputs from LLMs for a broader validation workflow.
Minimal recovery pseudocode
deadline = now() + total_budget
for attempt in allowed_attempts:
remaining = deadline - now()
if remaining < minimum_useful_time:
break
result = call_provider(timeout=min(per_attempt_timeout, remaining))
if result.success and validate(result):
return result
if not is_retryable(result) or not operation_is_safe_to_repeat(result):
break
wait = retry_after(result) or exponential_backoff(attempt)
sleep(min(wait, deadline - now()))
return try_fallback_or_show_recoverable_error()
In production, separate transport errors, provider errors, validation errors, and business-logic errors so dashboards and alerts lead to the right owner.
When to recalculate
Revisit the runbook whenever an input changes. That includes provider pricing, published rate limits, retry headers, SDK timeout behavior, model availability, context limits, streaming semantics, or the capabilities of a fallback model. Recalculate after changing prompts, retrieval volume, tool usage, response schemas, or user-facing latency targets because each can alter duration and failure probability.
Also recalculate after an incident. Compare estimated and observed values for timeout rate, retry rate, attempts per turn, fallback success, incomplete streams, validation failures, and user-visible recovery time. If retries are frequent, the solution may be capacity planning, queueing, caching, routing, or a provider change rather than more retries. The LLM API Cost Optimization Guide can help connect retry behavior with token and routing costs.
Before each major release, run a failure-path test matrix:
- connection failure and DNS failure;
- per-attempt timeout and total deadline expiry;
- rate limit with and without retry guidance;
- provider outage or unavailable model;
- authentication and malformed-request errors;
- interrupted stream and duplicate-safe retry;
- empty, truncated, refused, and schema-invalid responses;
- fallback failure and circuit-breaker recovery.
Finish by checking the user message for every path. It should state whether the request was completed, whether the user can retry, whether their input was preserved, and whether an action may have happened despite the error. A quiet, bounded recovery path is usually more trustworthy than an apparently seamless response that hides uncertainty.
For the final implementation checklist, pair this runbook with the AI Chatbot Deployment Checklist, the AI Chatbot Evaluation Guide, and the AI Chatbot Security Checklist. Keep the calculations, provider assumptions, and failure tests versioned so the procedure remains useful as your assistant changes.