SaveMyToken logoSaveMyToken.Coding Plan
All knowledge base guides

BEGINNER LESSON 05 OF 6

RAG Top K Should Be Chosen by Evidence Coverage and Final Answer Quality

Compare one returned passage with three while keeping documents and questions fixed.

About 8 min to follow along · By West Mao and Alex George · Documentation reviewed 2026-10-05 · Build in your own stack · Updated

TL;DR

Run the same questions against the same documents with K set to one and three, including a question that needs two facts. Compare whether the returned evidence is complete and the final answer is supported, alongside token use and latency. More passages are useful only when the additional context improves the result enough to justify its cost and distractions.

WHAT YOU WILL MAKE

A small paired record of evidence coverage, answer support, latency, and token use.

Before you start

  • A retriever with a configurable result limit and at least one question needing two facts.
Previous lesson: RAG Debugging Starts by Tracing Where the Required Fact Disappears

Fictional practice files. Expected answers below come from these files; actual model responses may differ.

STEP 01

Write a two-fact question

OpenYour test set

Use the supplied FAQ or your own verified documents.

  1. Choose a question that needs both a booking limit and a printer location.
  2. Mark the two supporting passages.
  3. Keep a covered one-fact question and an uncovered question as controls.
Two-fact question
How long can I book a Harbor Studio meeting room, and where is the printer?

You should see

The expected facts and source passages are recorded before tuning.

If it does not work

If both facts live in one passage, choose a different pair so result count can matter.

STEP 02

Compare K = 1 and K = 3

OpenYour retrieval function

Run the same query with two result limits.

  1. Capture the ordered passage IDs for K = 1.
  2. Change only the result limit to K = 3 and repeat.
  3. Check whether both required passages appear. Do not infer quality from a higher count alone.
You should see

You can see whether the extra results add necessary evidence.

If it does not work

If ranking changes between runs, freeze document versions, filters, and model settings.

STEP 03

Check the full answer

OpenYour RAG endpoint and comparison worksheet

A better result list must still produce a supported answer.

  1. Run the complete answer path for both limits.
  2. Review citations against passages and record measured latency and tokens if available.
  3. Keep the setting only if held-out questions benefit without unacceptable noise, time, or cost.
You should see

The choice is backed by paired observations, with missing measurements left blank.

If it does not work

If retrieval improves but the answer does not, inspect context assembly and generation.

CONTINUE WITH LESSON 6Current RAG Answers Need New Evidence Indexed and Old Versions Retired

Replace a 60-minute rule with 90 minutes and remove the old version from searchable results.