Finding premature exits in the saved agent traces

After finishing run-record saving, I found that a run could end as Submitted without executing a command. In one saved run, the model described inspecting code, making a change and verifying it, but its only action block contained exit. The captured patch was empty.

I added per-call records for output item types and token usage, then kept each message’s phase and text together before joining the text for parsing. This lets me distinguish the model’s commentary from its final answer without losing which message contained a command. Missing usage remains unknown rather than being recorded as zero.

Claude helped me work through lists of dictionaries, generator expressions and joining strings, and ran checks using a substitute API client. I implemented the changes and ran the real agent. The saved calls showed several messages describing work before the environment had executed anything.

I added a guard that rejects exit before a command has been attempted, and revised the prompt to explain that only the returned action block is executed. The guard does not prove a task is complete. It prevents this specific way of stopping before work begins.

The latest saved run still attempted to exit on its first call, and the guard rejected it. That matters when interpreting the prompt change: the run continued because the check intervened, not because the unwanted response had disappeared.

The latest trace includes a valid command in a commentary message followed by an empty final answer. Later, the same command appears in both phases and is rejected by the existing one-block rule. Reading only final-answer messages would lose the valid command, while accepting duplicate blocks would be a separate behavior change.

For now the parser still requires exactly one block. Keeping the message boundaries makes the next decision evidence-based rather than assuming every format error has the same cause.

The latest saved run on sympy__sympy-15599 took about 100 seconds and 15 steps, ended as Submitted, and retained a patch. Its execution output includes a run of sympy/core/tests/test_arit.py reporting 82 passed and 3 expected failures.

That is useful evidence from one task and one test file. The patch has not yet been scored through the SWE-bench evaluation harness, and there is no complete benchmark result. Automatic scoring is the next milestone. Preserving a full trajectory when an unexpected exception interrupts the runner also remains unfinished.