Scoring the first saved agent patch

I added a separate command to score saved agent patches with the SWE-bench evaluation harness. Previously, I could inspect the agent’s own test output, but I had not checked its patch through the benchmark’s evaluation process. The first scored attempt now has that result.

The new score_run.py command reads an existing run folder and invokes the harness without calling the language model again. It keeps the harness summary, raw test logs and my scores.json summary beside the original trajectory and patch. This lets me revisit an earlier attempt without generating a new one.

I worked through the scoring implementation with Claude providing scaffolding, explanations, review and checks. One bug read the test-result keys with the wrong capitalization. Error handling then hid the mismatch and made a resolved attempt appear to have no test results. Reading the actual report structure was necessary to fix that.

The score record distinguishes resolved, unresolved, empty patch, failed patch capture and evaluation error. An empty patch means there were no retained changes. Failed capture means I do not have the patch. An evaluation error is different from a completed evaluation that did not resolve the task.

I also record the scorer version and evaluation data source. The installed harness needed fields absent from the earlier data source, so I switched to SWE-bench/SWE-bench_Verified and added an explicit override for scoring older runs. The evaluated source is recorded rather than silently changing the old run’s metadata.

The saved patch from the October 20 local-time run on sympy__sympy-15599 was evaluated with swebench 5.0.2. Its per-instance report says the patch applied successfully and the instance was resolved: the one FAIL_TO_PASS test passed, and all 80 PASS_TO_PASS tests passed. These are the benchmark’s repair and regression checks for this instance.

This is stronger evidence than the earlier test-file output, but it is still one selected attempt on a task I have used repeatedly while developing the agent. It is not a complete benchmark run or an estimate of performance on unseen tasks.

Claude finished scoring the other saved attempts. Across 13 attempts on sympy__sympy-15599, four were resolved, four were unresolved and five had empty patches. Each resolved attempt passed the one repair test and all 80 regression tests. Each unresolved attempt failed the repair test but passed all 80 regression tests. The empty patches were not test evaluations.

This corrects the earlier suspicion about regressions: the unresolved patches did not fully repair the issue, rather than failing the designated regression tests. Claude’s keyword search found test references in the resolved trajectories and not in the unresolved ones. That is a clue for further inspection, not proof that test execution caused success. These development attempts used evolving settings on one familiar task, so their counts are not a benchmark success rate. Broader batch runs, complete failure accounting and retaining infrastructure-failure labels in the compact summary remain ahead.