Build a retry queue that respects Retry-After and action identity
Schedule bounded retries from structured errors and HTTP delay hints without creating a burst of replacement actions.
Recover from provider unavailability without duplicating pending work
Triage a provider incident by capability, admission, known outcomes and cleanup before releasing a queued backlog.
Confirm the failing boundary
Begin with the actual structured error and affected operation stage. A missing provider configuration, permission denial, budget rejection and upstream outage can all prevent work, but they need different repairs. Authentication success only identifies the key; it does not establish provider readiness or admission.
Check a bounded set of authorized read-only diagnostics and the deployment’s expected capabilities. Do not create a phone number, send a message or allocate a resource merely to see whether the provider is alive. A live write probe can add cost and another uncertain operation during the incident you are trying to understand.
Pause new effects while preserving recovery
Stop automatic dispatch for the affected product or operation class and leave its queued intent records intact. Retain saved request identities, last known phases and deadlines. Allow safe status inspection, known-result recovery and supported cleanup so the pause does not prevent operators from resolving existing work.
Distinguish jobs never dispatched from attempts with unknown outcomes and actions whose provider acceptance is known. A single failed bucket is insufficient. An unknown attempt must not be resubmitted as new work when the provider returns, while a known accepted action may need only local projection recovery.
Control retries and resource lifetimes
Honor server retry hints and use bounded queue backoff. Coordinate aggregate concurrency so many workers do not simultaneously flood a recovering provider. Keep workflow expiration visible; a task whose user deadline passed should not suddenly execute hours later just because connectivity recovered.
Inspect running resources separately from pending creates. Provider unavailability does not necessarily stop runtime or recurring charges. Follow the supported stop or hangup path when authorized, preserve uncertain cleanup outcomes and verify terminal state when evidence becomes available. A spending limit does not silently terminate existing resources.
Reconcile before releasing the backlog
Recover known accepted outcomes and identify definitive rejections first. For genuinely unknown attempts, use the original operation identity and provider references rather than new keys. Check usage reservations alongside resource state so restoring availability does not leave stale commitments or duplicate settled quantities.
A fallback provider is a product decision with its own scope, pricing, authorization and data handling. Do not automatically reroute private content or bypass account controls during an outage. If an approved fallback exists, apply it only to work whose original outcome is sufficiently known and whose user intent still permits that effect.
Resume gradually and close the incident factually
Release a small eligible portion of the queue and observe admission, provider outcomes, journal persistence and cleanup. Separate the count of successful HTTP requests from completed business actions. Continue to track unresolved operations even after new requests succeed; recovery of availability is not resolution of every earlier effect.
Write the incident record with affected products, known time boundaries, unresolved categories and corrective work. Exclude credentials and private customer content. Note whether evidence was partial because tracing or provider reads were unavailable. A factual account supports follow-up improvements without inventing performance statistics or claiming that every queued item was safely retried.
Sources & verification
Reviewed against the checked-in Lupin implementation and official reference documentation on 2026-11-05. Operational examples are illustrative; no live paid action, provider mutation or external message was executed.
- Lupin error reference Checked 2026-11-05
- Lupin usage ledger and spending limits Checked 2026-11-05
- Lupin API rate limits Checked 2026-11-05
Prepared with AI assistance by Lupin editorial. How we research and maintain these guides.