Can I return these shoes?

Start with the shop's applicable rule, then write the reply. Sounding certain is not the same as knowing what this customer is entitled to.

The shop manager posts a short, dated notice covering this sale. It names these shoes and says they can be returned within 14 days. The AI search still shows the old rule. The customer has their receipt from seven days ago.

How would you answer this customer now?

• Show the new notice and explain the 14-day rule
  Why this choice?

  The manager's notice covers these shoes and this purchase date. You can answer directly from it; there is no need to make the AI rewrite one clear sentence.

• Use the AI search result because it checks more documents
  Why this choice?

  More documents do not help if the answer misses today's notice. The old search result is not the current rule for this sale.

• Ask the customer to wait until AI search catches up
  Why this choice?

  Waiting would make sense if the rule were unclear. Here, the manager has already supplied a clear answer for this purchase, so waiting adds work without resolving anything.

Find the rule that applies, not just an answer that sounds right.

Where would you look if a confident answer disagreed with the shop's latest notice?

Grounding means tying an answer to a source you can check. Retrieval finds relevant source material for the AI. If that search is out of date, a current source may be more useful on its own.

Choose how a support assistant finds current evidence, then respond when the evidence service cannot keep up.

Customer support · 8 minutes

The problem

Synthetic practice scenario: all quantities and evaluation outcomes are illustrative, not reported research. A service team wants draft answers about returns across product lines; staff currently search policy documents and must verify the policy before replying.

Your initial decision

How should the first assistant help staff answer a policy question?

• The illustrative corpus contains 600 approved documents with product, region, effective-date and access metadata.
• Questions usually concern one product and region; exact product codes and natural-language descriptions both matter.
• Policy updates arrive daily; the assistant may draft replies but cannot approve exceptions or issue refunds.
• Unsupported or conflicting evidence must produce a handoff, not an invented policy.

Compare the alternatives

• Offer a product-and-region source menu

  A product-and-region lookup is easy to audit but leaves staff to read and reconcile long documents.

• Draft from filtered retrieved passages

  Combine keyword and semantic retrieval with access and effective-date filters; ranking needs tuning and freshness monitoring.

• Put complete permitted policy documents in context

  Full-document prompting preserves surrounding exceptions and avoids chunk tuning, but broad product queries create large contexts that must be refreshed and access-filtered.

Mechanism, trade-offs, and evidence

• Start with the task: staff need the currently applicable rule, not the most fluent paragraph; compare against their existing lookup workflow.
• Lexical search preserves exact codes while semantic search helps paraphrases, but neither establishes access or policy validity without metadata filters.
• Evaluate retrieval misses separately from unsupported generation and include conflicts and unanswerable questions in a held-out task set.
• A graph is not justified by mostly local policy questions; introduce relationship retrieval only if a distinct multi-document failure appears.

Reconsider after a change

During a policy migration, the index is stale and new effective-date mappings are not yet validated; the pilot is temporarily narrowed to a signed emergency bulletin.

What should staff use until the migration is verified?

• In this synthetic phase, only 12 short bulletin entries are in scope and staff can select the relevant entry directly.
• The index has a 48-hour update lag, exceeding the illustrative same-day freshness requirement.
• The signed bulletin is the temporary authority; wider policy questions require a human policy owner.
• The migration needs access and effective-date regression checks before normal retrieval resumes.

Compare the alternatives

• Restore the filtered retrieval pipeline

  Restoring the filtered pipeline can recover broad coverage but requires verified mappings and freshness monitoring before release.

• Generate answers from the complete signed bulletin

  Passing the bulletin directly avoids the stale index and preserves its full context, but generated synthesis adds verification work to a small set of directly selectable entries.

• Return the selected bulletin entry directly

  A deterministic selector returns the signed excerpt and source link immediately; staff must select the region and confirm coverage, while cross-policy questions still require a handoff.

Mechanism, trade-offs, and evidence

• The original architecture was conditional on fresh, valid evidence; that assumption has changed even though the model has not.
• A narrower deterministic source now meets the immediate need with less uncertainty, so reducing complexity is a justified adaptation.
• Record deferred questions and migration failures to guide restoration, rather than silently treating reduced coverage as full service.
• After validation, compare the restored pipeline with direct lookup again; the former recommendation is not a permanent commitment.

Related technical concepts

Follow-up lessons retain their existing sign-in and plan requirements.

• rag basics
• retrieval engineering
• hallucination grounding
• ai ready data
• context engineering
• graph rag
• agent uncertainty calibration
• data knowledge operations

Technical sources

• Introducing Contextual Retrieval — Anthropic Engineering

  Explains chunk context, lexical and semantic retrieval, and reranking; vendor results are not evidence about these fictional support documents.

• Demystifying evals for AI agents — Anthropic Engineering

  Explains tasks, graders, repeated trials and outcome evaluation for agents; use task-specific tests rather than importing reported scores.

• Building effective agents — Anthropic Engineering

  Distinguishes fixed workflows from model-directed agents and recommends testing simpler solutions first; its tooling examples are historical.

Explore the technical decision