Using an AI agent on an existing application: what evidence should you request?
Match the review to the risk of the change, reproduce the defect, then check that a test fails before the fix and passes afterwards.
Ask what the change allows you to verify
An agent proposes an export fix and reports that the tests pass. The review question remains: do the tests cover the reported defect? I suggest requesting evidence proportionate to the risk and tied to the expected behaviour. This method is a proposed way of working, not a comparative result on agents.
In a fictional scenario, the export omits cancelled orders that a manager wants to view. First decide whether including them is part of the requirement. The agent should not settle that rule alone based on the existing code.
A short evidence record
- Context: approved business rule, initial state and environment where the defect occurs.
- Diff: changed files, the purpose of each change and justification for any moves.
- Reproduction: fictional data and an observable result before the fix.
- Validation: a command or journey that fails before and passes afterwards; other uses that may be affected.
- Limitations: checks not run, missing access and delivery uncertainties.
Keep command output alongside the version tested. A list of commands in a response does not prove they were run. An added test that merely repeats the new implementation may pass with the defect present; replaying it on the defective state checks its ability to detect the problem.
Avoid an evidence record that costs more than the change
A spelling correction in a caption deserves proofreading and a rendering check. It does not require a load-testing lab. Conversely, a new authorisation rule requires checking denied access and data ownership. Diff size alone does not measure that risk.
GitHub's recommendations for its agent emphasise bounded tasks and human review. They do not guarantee a correct output. The reviewer must understand the critical behaviour and be able to challenge the proposal, even when automated checks are green.
A possible small experiment
Choose an already reproducible defect and fix the acceptance criterion before making the request. Give the agent a limited correction, retain its iterations, then have the diff reviewed and the scenario replayed. Record what the evidence establishes and what remains unknown.
This experiment has not been carried out with a group of reviewers here. No time saving or error rate is inferred from it. Its value would lie in comparing the questions still open at review: if the record lets someone find the defect and verify the correction, it makes acceptance more explicit.