Yousuf Golding
← All work notes

Project overview · In progress

MP1 · Minimal software-engineering agent

A tutorial-based agent implementation, developed step by step toward a complete SWE-bench evaluation.

Last updated

Based on the minimal-agent tutorial and mini-swe-agent reference. I implement the code with Claude providing scaffolding, explanations, review, and some checks.

Goal

Build and understand my own agent, then evaluate it across a chosen SWE-bench collection, accounting for successful attempts, unsuccessful attempts, and execution failures.

Implemented

An agent runner with execution environments, patch and response records, unexpected-error recording, and a separate SWE-bench scoring command. Batch execution is currently a skeleton, not an implemented feature.

Results so far

Scoring is complete for 13 saved attempts on one development task: 4 resolved, 4 unresolved and 5 empty patches. All 8 evaluated patches passed 80/80 regression tests; only the 4 resolved patches passed the repair test. These are repeated development attempts, not a complete benchmark result.

Next milestone

Implement and check the sequential batch runner and planned-instance metadata. Extend scoring to preserve agent-error and infrastructure-failure distinctions before broader evaluation.

Project history

Dated accounts of the work. Earlier notes describe the project at that point in time.

  1. Keeping the record when an agent run fails

    Preserving failed attempts and partial trajectories, then preparing the next step toward batch execution.

  2. Scoring the first saved agent patch

    Adding a separate evaluation command, preserving distinct failure states, and getting the first resolved result from the SWE-bench harness.

  3. Finding premature exits in the saved agent traces

    Recording individual model messages, guarding against an exit before execution, and checking what a single-task result actually establishes.

  4. Keeping the patch and the record of a run

    Capturing changes before container cleanup, saving trajectories, and finding a failure in the final summary.

  5. Moving run settings into one configuration

    Loading configuration once, keeping prompt values consistent with execution, and separating build history from current code.

  6. Building the execution environment for my minimal agent

    Separating command execution from the agent loop, working through timeouts and cleanup, and checking the single-task runner.