Outcome checks
Define what a useful answer must contain. An optional judge on another model checks the answer against that rubric and can ask the agent to try again.
Read the docsFind out if your agent got better.
Change the instructions, model or tools. Test the proposed agent against the installed version on the same tasks, then compare useful results and model costs.
Illustrative incident summary. Results and model costs below are invented.
| What you asked for | Installed | Proposed |
|---|---|---|
| Name the affected service | Present | Present |
| Cite the supporting logs | Missing | Present |
| Suggest a next step | Present | Present |
- Installed model cost
- $0.02
- Proposed model cost
- $0.03
Evidence improves; model cost rises from $0.02 to $0.03. Is that useful improvement worth the extra cost?
Choose changes that improve useful results at a cost your team accepts. Check more than one task.
What you can do
Compare changes
Run both versions on the same test cases. Supported comparisons replace writes with simulated results; unsupported agents are skipped and reported.
Read the docsPlugin improver
Let an agent propose a fix to a plugin you own. A person reviews the change before a draft pull request opens, and a person decides whether to merge it.
Read the docsDid the change help your team?
Assess these on the same representative tasks. Combine job records with your team’s review; these are evaluation questions, not an automatic score.
| Measure | What to check |
|---|---|
| Useful results | Did a person accept the result? Did it solve the original task? |
| Cost per useful result | Include retries and rejected attempts when comparing model costs. |
| Human effort | How often did someone need to approve, correct or finish the work? |
| Recovery | After an interruption, what resumed, restarted or needed intervention? |