SaveMyToken logoSaveMyToken.Coding Plan
← RAG & retrieval

FREE AI FDE PRACTICE / RAG & RETRIEVAL

RAG Can Cite the Right Document While Missing the Decisive Exception

By West Mao and Alex GeorgeIntermediate 8-min practiceUpdated Editorial review: 2026-10-05

TL;DR — Approach in brief

A policy assistant cites the right handbook while missing an exception that changes the answer. Practice tracing the decisive evidence through extraction, chunking, retrieval and the final model input. Explain how you would isolate the failing stage and judge claim support. Try the diagnosis before opening the reference; citation presence by itself does not resolve the case.

YOUR SCENARIO

How would you approach this?

A policy assistant cites the correct handbook but tells a customer that opened products can be returned. The handbook contains an exception excluding them. How would you isolate the failure?

This is an illustrative practice scenario. State any additional assumptions in your answer.

Make your case first.

Clarify the goal, identify the biggest uncertainty, outline an approach, and explain how you would test it. Spend about 8 minutes before opening the reference.

Your notes are not submitted or saved. Keep a copy before leaving this page.

Reveal reference approach Clarifying questions, decisions, and tradeoffs

Clarify before designing.

  • Was the exception in the exact context sent to the model, or only somewhere in the source document?
  • Which source version, query, retrieval settings, and prompt produced the answer?

One defensible approach

  1. 01

    Inspect the actual evidence path

    Reproduce the case with a redacted trace. Follow the exception through extraction, chunking, candidate retrieval, reranking, and final context assembly. A correct document ID does not establish that the decisive sentence reached generation.

  2. 02

    Test the failing stage

    If the exception is absent, inspect chunk boundaries and candidate coverage. If present, test whether the answer preserves both the rule and exception with reduced distracting context. Change one factor at a time and retain the original failure as a regression case.

  3. 03

    Judge claims, not citation presence

    Check whether each material claim follows from the accessible passage. Include negative examples with valid citations but false statements. Review abstention when the policy cannot support an answer.

Explain the tradeoff

More retrieved text can recover an exception but can also add conflicting policy versions. Measure evidence completeness and answer correctness together.

Common mistakes

  • Assuming a citation proves the answer is supported.
  • Changing the model before locating the missing evidence.

KEEP THE CONVERSATION GOING

Try the follow-ups.

  1. How would your diagnosis change if the document is a scanned table?
  2. What regression test would catch this before a release?

Review your own answer.

Tick the points you covered. This is a reflection checklist, not an automated score or a hiring prediction.

Check the underlying concepts.

The scenario and reference approach were written for SaveMyToken. These sources support the technical concepts; they do not report this question being asked by an employer.

LangChain: Retrieval ↗Anthropic: Demystifying evals for AI agents ↗