YOUR SCENARIO
How would you approach this?
Your app sends a long policy and tool definitions on every request. Caching was enabled, but billed usage barely changed. Each request also includes a timestamp and user-specific settings near the beginning. How do you investigate?
This is an illustrative practice scenario. State any additional assumptions in your answer.
Make your case first.
Clarify the goal, identify the biggest uncertainty, outline an approach, and explain how you would test it. Spend about 8 minutes before opening the reference.
Your notes are not submitted or saved. Keep a copy before leaving this page.
Reveal reference approach Clarifying questions, decisions, and tradeoffs
Clarify before designing.
- What does this provider cache, and what are its eligibility, lifetime, write, and read billing rules?
- Do actual request logs and usage fields show reuse for the same eligible prefix?
One defensible approach
- 01
Inspect the request shape
Compare consecutive serialized requests. Look for changing timestamps, tool order, formatting, and user data inside the intended reusable prefix. Confirm the endpoint and model support the requested caching behavior.
- 02
Separate stable and variable content
Place eligible shared instructions and tool definitions in a stable order, with per-request content afterward where the provider's rules permit. Keep version changes deliberate. Do not move sensitive data across users merely to improve reuse.
- 03
Measure warm and cold behavior
Run controlled repeated requests within the relevant cache lifetime. Inspect cache-specific usage and compare total billed reads, writes, and uncached tokens. Report results for this workload rather than assuming a universal discount.
Explain the tradeoff
Maintaining a stable prefix can improve reuse but must not freeze a policy that needs updating. Correctness and version invalidation remain part of the design.
Common mistakes
- Confusing prompt-prefix caching with reusing a previous answer.
- Comparing only the second request while ignoring cache creation cost.
KEEP THE CONVERSATION GOING
Try the follow-ups.
- How do frequent policy updates change the economics?
- Would a semantic response cache solve the same problem?
Review your own answer.
Tick the points you covered. This is a reflection checklist, not an automated score or a hiring prediction.
Check the underlying concepts.
The scenario and reference approach were written for SaveMyToken. These sources support the technical concepts; they do not report this question being asked by an employer.
Anthropic: Prompt caching ↗