YOUR SCENARIO
How would you approach this?
A service company offers 2,000 scanned work orders for an extraction pilot. Some scans are unreadable, several templates changed, and the customer cannot provide a complete ground-truth spreadsheet. How do you plan the first week?
This is an illustrative practice scenario. State any additional assumptions in your answer.
Make your case first.
Clarify the goal, identify the biggest uncertainty, outline an approach, and explain how you would test it. Spend about 8 minutes before opening the reference.
Your notes are not submitted or saved. Keep a copy before leaving this page.
Reveal reference approach Clarifying questions, decisions, and tradeoffs
Clarify before designing.
- Which fields drive a downstream decision, and which errors require review?
- Who can label a representative sample and resolve contradictory records?
One defensible approach
- 01
Inspect before promising coverage
Sample across templates, dates, scan quality, and common exceptions. Track whether errors come from missing source data, extraction, or interpretation. Ask for the minimum authorized data needed to test those categories.
- 02
Build a small reviewed set
Have domain staff label the required fields and mark genuinely ambiguous cases. Reserve a separate sample for validation. Report coverage alongside field accuracy so discarding difficult documents cannot look like an improvement.
- 03
Deliver with a review queue
Start with supported templates and show extracted values beside source locations. Route unsupported or ambiguous records to operators. Quantify review effort and maintain a backlog of source-data fixes with customer owners.
Explain the tradeoff
Restricting the supported document set can make the pilot useful sooner, but the customer must see how much work remains outside that boundary.
Common mistakes
- Treating an absent source field as a model error.
- Evaluating only the clean documents used during development.
KEEP THE CONVERSATION GOING
Try the follow-ups.
- How do you report success when operators correct many outputs?
- A new template arrives tomorrow. What happens?
Review your own answer.
Tick the points you covered. This is a reflection checklist, not an automated score or a hiring prediction.
Check the underlying concepts.
The scenario and reference approach were written for SaveMyToken. These sources support the technical concepts; they do not report this question being asked by an employer.
Google: Rules of Machine Learning — objectives and metrics ↗Anthropic: Demystifying evals for AI agents ↗