Keeping the record when an agent run fails

I started the next stage by addressing what happens when an agent run crashes. Previously, an unexpected exception could leave only the run configuration on disk, without the conversation or model-call records needed to understand the failure.

The runner now records an Error outcome with the exception type, message and traceback, alongside the conversation and model-call records collected so far. If the environment started, it attempts to capture the partial patch before cleanup.

The partial patch stays in the trajectory for diagnosis, but the prediction sent for scoring contains no patch and includes an agent_error marker. A crashed attempt should remain visible in the run record without its unfinished changes being treated as a submitted solution. The scoring summary still needs to interpret that marker explicitly.

The error boundary covers environment startup, agent execution and patch capture. Invalid instance IDs and configuration errors encountered before that boundary still propagate. Repeating a setup mistake across a batch would not produce useful evidence. Keyboard interrupts also remain able to stop execution.

I implemented this with Claude providing guidance and review. One bug stored the Exception class instead of the raised exception’s name, which made the record impossible to serialize as JSON. Working through it helped separate the exception object, its type and the text that belongs in a saved record.

Claude recorded four controlled checks using Docker and a substitute agent, without model API calls: an edit followed by a ConnectionError, failed environment startup, a normal edit, and failed patch capture. The crash record retained a partial patch and error details, while the prediction excluded the patch. The checks reported cleanup in each case.

These checks exercise specific failure paths. They do not establish that every possible failure is recoverable, and they do not add new benchmark performance results.

Claude prepared a sequential batch-runner skeleton and a shared run-and-save helper for me to implement. The intended run metadata will list every requested instance, so an interrupted batch can distinguish work that was never reached from work that was never requested.

The skeleton still contains NotImplementedError placeholders. Batch execution and planned-instance recording are not working features yet. The completed scoring comparison remains 13 attempts on one SymPy task, with four resolved, four unresolved and five empty patches.