[ HARD ]
Build an eval harness for a refund-approval agent
Quick answer
Brief: An agent approves or escalates refund requests. A wrong approval costs money; a wrong escalation costs time. Build the harness that decides whether a new version can ship.
Tasks
- 1. Design the labelled dataset and its edge cases
- 2. Weight errors by their cost
- 3. Block a release that regresses on the costly errors
Show hints
- → Report approval errors separately from escalation errors
- → Include adversarial requests in the set
Skills