The shop website will not take orders

Help the responsible person see what is happening. Keep the alert moving, and keep live website changes separate from the AI's suggestions.

The website owner checks the failed orders and finds a new checkout setting is responsible. They have a tested way to restore the previous setting and approve doing it now. The AI is still suggesting a full restart.

What should the team do with the AI's suggestion now?

• Have the owner restore the checked setting and watch checkout
  Why this choice?

  The owner has checked the cause and approved a known repair. They can make the change while the AI helps watch whether checkout recovers. Record what happened and keep watching for failed orders; the repair still needs confirmation.

• Delay the repair until the AI agrees with the owner
  Why this choice?

  The owner has information the AI suggestion does not reflect. Waiting for agreement adds delay; use the checked repair and confirm whether customers can place orders again.

• Let the AI restart everything as well
  Why this choice?

  Approval to restore one setting is not approval for a full restart. Adding another change can interrupt orders and make it harder to tell which action helped.

A useful diagnosis helps someone act; it does not give the AI permission to act.

Which details would help the website owner decide, and which action should still need their approval?

Read-only assistance lets an AI inspect approved service information and summarize problems without changing the live system. Permission to diagnose and permission to repair are separate, even when a diagnosis looks convincing.

Choose an incident-assistance loop that reduces operator load without turning uncertain diagnosis into production writes.

Service operations · 8 minutes

The problem

Synthetic practice scenario: all quantities and evaluation outcomes are illustrative, not reported research. An on-call team receives noisy service alerts and wants an agent to assemble useful evidence before an operator chooses a response.

Your initial decision

How should the agent assist when an alert fires?

• User-visible error and latency signals determine paging; an agent must not suppress a critical page.
• The approved runbook identifies read-only metrics, recent deployment metadata and sanitized trace summaries.
• Production mutation is reserved for the incident commander; customer payloads and secrets are excluded from the evidence packet.
• An illustrative 60-second collection budget limits repeated tool calls before the agent hands off what it has.

Compare the alternatives

• Improve the runbook and deterministic dashboard

  A deterministic view is predictable and cheap, though it may not summarize evidence across varied incident narratives.

• Collect a bounded read-only evidence packet

  A fixed collection workflow plus source-linked summarization reduces lookup work, but the operator still diagnoses and approves intervention.

• Let a read-only agent choose diagnostic queries

  An adaptive investigator can pursue unusual symptoms within the same tool and time limits, but variable query sequences may exhaust the budget before collecting basic evidence.

Mechanism, trade-offs, and evidence

• Keep alerting anchored to user-visible symptoms rather than generated confidence; the assistant must not become a new single point of paging failure.
• Use trace IDs and artifact versions to connect observations, with payload minimization and appropriate telemetry access.
• Require each suggested cause to cite a signal and distinguish observation from inference.
• Measure operator time to useful evidence, misleading summaries and collection failures against the existing dashboard.

Reconsider after a change

A candidate model ranks causes more accurately in incident replays, but the next rollout introduces an untested failover topology.

What should the higher diagnostic score change during the rollout?

• A vendor reports a higher diagnostic benchmark score; in an illustrative local held-out replay, correct top-ranked causes rise from 72 of 100 to 86 of 100 using fixed evidence packets, not adaptive query loops.
• The new topology changes dependency and failover behavior, so historical repair sequences may be invalid.
• Mutation authority still belongs to the incident commander and the paging contract is unchanged.
• The team can add topology metadata and run recovery drills outside production.

Compare the alternatives

• Switch to a bounded adaptive diagnostic loop

  The improved model might discover new dependencies through read-only queries, but packet-based replay scores do not test query selection, budget exhaustion or the changed topology.

• Use a topology-aware dashboard without summaries

  Direct dependency and symptom views avoid generated diagnostic errors, but operators resume the cross-service synthesis that the packet workflow had reduced.

• Keep the bounded packet and add topology metadata

  The candidate improves cause ranking inside the tested collection workflow, while topology checks and operator diagnosis remain necessary; missing dependencies are surfaced rather than inferred.

Mechanism, trade-offs, and evidence

• The recommendation stays advisory even with local improvement: packet-based cause ranking improved, but adaptive investigation and repair remain untested in the new topology.
• Inspect the remaining misdiagnoses and model-version traces rather than relying solely on the improved average.
• Re-evaluate the evidence collector on the new topology and keep incomplete packets visibly incomplete.
• Promotion to remediation would need separately demonstrated recovery behavior and explicit authorization, not a higher diagnostic threshold alone.

Related technical concepts

Follow-up lessons retain their existing sign-in and plan requirements.

• agent observability
• agent troubleshooting
• agent ops
• loop engineering
• proactive agent design
• agentic automation thresholds

Technical sources

• Monitoring Distributed Systems — Google SRE

  Connects user-visible symptoms, useful alerting and diagnostic signals; monitoring supports but does not authorize remediation.

• Semantic Conventions for Generative AI — OpenTelemetry

  Defines GenAI spans, metrics and events in the successor repository; pin a supported version and avoid collecting sensitive payloads by default.

• Demystifying evals for AI agents — Anthropic Engineering

  Explains tasks, graders, repeated trials and outcome evaluation for agents; use task-specific tests rather than importing reported scores.

Explore the technical decision