YOUR SCENARIO
How would you approach this?
A policy assistant cites the correct handbook but tells a customer that opened products can be returned. The handbook contains an exception excluding them. How would you isolate the failure?
This is an illustrative practice scenario. State any additional assumptions in your answer.
Make your case first.
Clarify the goal, identify the biggest uncertainty, outline an approach, and explain how you would test it. Spend about 8 minutes before opening the reference.
Your notes are not submitted or saved. Keep a copy before leaving this page.
Reveal reference approach Clarifying questions, decisions, and tradeoffs
Clarify before designing.
- Was the exception in the exact context sent to the model, or only somewhere in the source document?
- Which source version, query, retrieval settings, and prompt produced the answer?
One defensible approach
- 01
Inspect the actual evidence path
Reproduce the case with a redacted trace. Follow the exception through extraction, chunking, candidate retrieval, reranking, and final context assembly. A correct document ID does not establish that the decisive sentence reached generation.
- 02
Test the failing stage
If the exception is absent, inspect chunk boundaries and candidate coverage. If present, test whether the answer preserves both the rule and exception with reduced distracting context. Change one factor at a time and retain the original failure as a regression case.
- 03
Judge claims, not citation presence
Check whether each material claim follows from the accessible passage. Include negative examples with valid citations but false statements. Review abstention when the policy cannot support an answer.
Explain the tradeoff
More retrieved text can recover an exception but can also add conflicting policy versions. Measure evidence completeness and answer correctness together.
Common mistakes
- Assuming a citation proves the answer is supported.
- Changing the model before locating the missing evidence.
KEEP THE CONVERSATION GOING
Try the follow-ups.
- How would your diagnosis change if the document is a scanned table?
- What regression test would catch this before a release?
Review your own answer.
Tick the points you covered. This is a reflection checklist, not an automated score or a hiring prediction.
Check the underlying concepts.
The scenario and reference approach were written for SaveMyToken. These sources support the technical concepts; they do not report this question being asked by an employer.
LangChain: Retrieval ↗Anthropic: Demystifying evals for AI agents ↗