Our online helper is down. How can we help visitors?

A backup should preserve people's trust and information boundaries, not just produce an answer.

The desk owner approves and tries a backup service for public questions about opening times and directions only. Private questions still stay with staff. If the backup also fails, the sheets and phone line remain available.

How should June use the backup next time?

• Send every question because the backup is approved
  Why this choice?

  The approval covers public opening times and directions, not personal cases. If June is unsure whether a question contains private details, keep it with staff rather than stretching the permission.

• Use it only for the approved public questions
  Why this choice?

  That restores useful help within the agreed boundary. Keep private or uncertain requests with staff, and return to the sheets or phone line if the backup cannot answer.

Choose a continuity plan that is both useful and permitted for the information involved.

Which questions could safely use a public backup, and which should stay with a person?

Routing checks data class, recipient approval and tested quality before switching providers. Bounded attempts prevent repeated failures; uncertain external actions require reconciliation rather than blind replay.

Maintain service during an outage without quietly moving private data or duplicating consequential actions.

Business continuity and provider outages · 9 minutes

The problem

Synthetic practice scenario: all quantities and evaluation outcomes are illustrative, not reported research. Fictional outcomes are not guarantees. A service desk relies on an external model to draft responses; it needs an honest continuity plan when that provider stops answering.

Your initial decision

What should the service do when its only approved provider is unavailable?

• In the fictional incident, the primary model has failed for 15 minutes and request queues are growing.
• Only that provider is approved for private customer case text; another reachable provider has no authorization for these records.
• Approved local response templates and staff handling remain available, including a route for urgent cases.
• The draft service cannot execute customer actions; any separate action with an unknown result must be reconciled, not replayed.

Compare the alternatives

• Queue requests for a bounded wait on the primary

  A short bounded queue may bridge a brief interruption without changing processors, but prolonged waiting can hide urgency and increase abandonment.

• Use local templates and a visible staff handoff

  Degraded service offers less tailored help but preserves approved data boundaries and gives customers a clear route while the provider recovers.

• Route drafts to the reachable secondary provider

  A secondary could restore drafting speed, but current reachability does not establish authorization, data handling terms or quality for these cases.

Mechanism, trade-offs, and evidence

• Continuity is a user-service problem, not a requirement to keep model calls flowing at any cost.
• Availability and permission are independent: a reachable endpoint is not an approved destination for customer information.
• Bound waits and expose degraded behavior; otherwise a technically orderly retry loop can leave customers without useful help.

Reconsider after a change

A continuity review approves a tested secondary for a defined non-sensitive request class; routing now separates that class from private cases and includes recovery limits.

What fallback should activate during the next primary outage?

• An independent illustrative replay of 180 non-sensitive draft requests meets the agreed quality and latency thresholds on the secondary.
• The data owner approves only the specified request class; private or uncertain classifications remain on local templates and staff handling.
• Routing has a deadline, a single allowed fallback attempt and budget caps; health checks prevent repeated switching between providers.
• The secondary produces drafts only; binding commitments still require authorized human approval and unknown action outcomes are never replayed.

Compare the alternatives

• Hold every request until the primary recovers

  Waiting avoids invoking the secondary, but needlessly delays the approved request class now that a tested and bounded continuity path exists.

• Keep local service for private or uncertain requests

  Templates and staff protect records outside the approval boundary, but offer less tailored help than the secondary for requests that do qualify.

• Fail over only the approved non-sensitive draft class

  A tested secondary restores useful drafting within the permitted boundary, while bounded attempts and local fallback limit cost and repeated failure.

Mechanism, trade-offs, and evidence

• The changed permission boundary and independent task evidence together justify fallback; neither alone would have been sufficient.
• Measure useful completed requests rather than successful HTTP calls, and retain a local mode if both providers fail.
• Provider switching applies to permitted draft computation, not blind replay of actions that may already have affected a customer.

Related technical concepts

Follow-up lessons retain their existing sign-in and plan requirements.

• model routing
• architecture platform operations
• agent observability
• agent security

Technical sources

• Monitoring Distributed Systems — Google SRE

  Connects user-visible symptoms, useful alerting and diagnostic signals; monitoring supports but does not authorize remediation.

• MCP Security Best Practices — Model Context Protocol

  Describes consent and authorization boundaries for tool integrations; adopting a protocol does not grant access to customer data or actions.

• Demystifying evals for AI agents — Anthropic Engineering

  Explains tasks, graders, repeated trials and outcome evaluation for agents; use task-specific tests rather than importing reported scores.

Explore the technical decision