Measure what Top K changes

WHAT YOU WILL BUILD

A three-row evaluation showing the tradeoff between evidence coverage and extra context.

The example question needs both a rule and an exception. Precision is relevant retrieved items divided by retrieved items; recall is relevant retrieved items divided by all labeled relevant items. They describe retrieval, not the truthfulness of a final answer.

FOLLOW ALONG

Work through the example.

1. 01

   Fix the relevance labels

   Mark rule and exception as relevant; shipping is irrelevant to this question. Do this before changing K, so the target cannot move with your result.

2. 02

   Run K from one to three

   At K=1 precision is 1.00 but recall is only 0.50. At K=3 recall reaches 1.00 while precision drops to 0.67 because shipping is included.

3. 03

   Check the denominator

   For K=2, only one of the two retrieved items is relevant, so precision is 1/2. There are two relevant items overall, so recall is also 1/2.

4. 04

   Build a small comparison sheet

   Add several representative questions with their relevant document IDs. Compare two configurations on the same questions. Record per-question failures before averaging, then review the answers and measure latency separately.

THE COMPLETE LAB

Run it locally.

Run this command from the folder containing your downloaded file:

node rag-evaluation.mjs

// SaveMyToken local lab. Run with Node.js 22 or newer.
import assert from "node:assert/strict";

const relevant = new Set(["rule", "exception"]);
const ranked = ["rule", "shipping", "exception"];
function evaluate(k) {
  const retrieved = [...new Set(ranked)].slice(0, k);
  const hits = retrieved.filter(id => relevant.has(id)).length;
  return { precision: hits / retrieved.length, recall: hits / relevant.size };
}
for (const k of [1, 2, 3]) {
  const result = evaluate(k);
  console.log("K=" + k + " precision=" + result.precision.toFixed(2) + " recall=" + result.recall.toFixed(2));
}
assert.equal(evaluate(1).recall, 0.5);
assert.equal(evaluate(3).recall, 1);
assert.equal(evaluate(3).precision, 2 / 3);

Expected output for the unchanged example

K=1 precision=1.00 recall=0.50
K=2 precision=0.50 recall=0.50
K=3 precision=0.67 recall=1.00

The assertions also check the baseline behavior. After changing an input, predict the result and update the relevant assertion.

EXPLAIN WHAT YOU LEARNED

Interview practice

Try answering aloud before opening the reference answer. These are original learning questions, not a record of any employer's interviews.