Six months of agent-assisted development in Counterfact

The hypothesis under evaluation

Agent-assisted development can increase delivery while preserving a working verification and repair loop.

The evaluation makes that proposition concrete with three checks. The matched-window PR count should exceed the highest observed earlier count. Each confirmed external defect in the seven-case population should have a traceable public correction, with its report-to-corrected-publication time shown. Each associated check should fail on the faulty behavior and pass on the correction. Falling short on any check would count against that part of the hypothesis.

This is a retrospective assessment of one project. It seeks evidence strong enough to inform the maintainer’s workflow and publishes the methods so other people can question the interpretation.

Workflow changes under evaluation

Counterfact turns an OpenAPI description into a stateful mock API. The maintainer began merging agent-authored product changes on March 10, 2026. During the same period, the project invested in how changes would be checked.

On March 26, the project’s black-box tests moved toward product-level journeys. On March 27, the maintainer added a required manual-acceptance checklist for qualifying changes. The journey tests were reworked into feature files and bindings in August. A September repair made the acceptance check work with the merge queue.

These changes document the approach under evaluation: use agents to help produce changes, and invest in independent ways to check the resulting behavior. Adding a gate does not prove it was effective. That is why this study follows actual reports and exercises actual checks.

1. Delivery moved substantially

The project merged 279 non-dependency PRs during the study period—3.7 times the highest observed count in the same calendar window of the previous four years.

Test declarations grew in every comparable cohort

The 2026 cohort added 634 explicit test declarations in the matched window. The earlier comparable cohorts added 10, 48, and 6. That is a large expansion in verification inventory alongside delivery, although a count does not show whether a check can detect an important failure.

2. The external repair loop was working

Seven confirmed external defect reports arrived after adoption. All seven concerned behavior already present in releases before adoption. Five reached a corrected release on the next calendar date; all seven within five calendar days. The study counts a repair when it reaches a public release, not when an issue closes.

There are only seven reports, and one person supplied six of them. They show that these observed failures were handled promptly. They do not measure everything users encountered, and they do not establish a lower defect rate.

3. The checks detect the behaviors they claim to test

Test declarations grew from 270 to 904, but a larger test inventory does not answer whether the tests are useful. Codex asked a more direct question: does the relevant check fail on the faulty implementation and pass on the correction?

Codex attempted that comparison for all seven external cases. Six use assertions added with the historical repairs, copied into affected environments where needed. The packaging case uses an audit-created check on published packages. These are later experiments; the study does not claim the tests caught these defects before users did.

7 of 7 comparisons distinguished affected from corrected behavior. For example, the reserved-word check fails when generated TypeScript uses an invalid operation ID and passes when the identifier is escaped. The package check shows that an obsolete patch hook runs in 2.11.0 and is absent in 2.12.0; both installations succeed.

This establishes something specific about these checks. It does not establish that the whole suite passed historically or that the suite covers every important failure. The experiment records contain the revisions, commands, adaptations, and output needed to challenge or repeat the result.

The failures matter to the conclusion

The broader public record contains 17 confirmed released defects reported after adoption, including the seven external reports. Discovery attribution is undocumented for the other 10. There are also 2 findings corrected before release and 4 confirmed behaviors whose release status remains unresolved. The case ledger keeps those distinctions visible.

One failure directly challenges any easy success story. A root-middleware repair shipped in 2.2.0 on March 19 and broke middleware behavior. A follow-up repair reached users in 2.3.0 on March 23. This was a new released regression during the agent-assisted period.

Release provenance also failed. When 2.16.2 appeared without the usual Git tag, GitHub release, or visible publishing provenance, a user asked whether it was legitimate. Patrick McElhaney confirmed that he had published it from his machine. That explanation does not replace the public evidence a user should be able to inspect.

These cases reinforce two requirements: verify the affected product journey, and make the path from reviewed source to published package visible. Faster change makes those responsibilities more consequential.

Conclusion and continuing checks

The evidence shows a meaningful increase in delivery, with concrete evidence that the repair loop and the fixed seven-case checks are working. Patrick McElhaney judged that result sufficient to continue this way of using AI. The evidence for increased delivery is stronger than the evidence about overall quality.

Future scrutiny should test whether these results persist as the software and its users exercise more paths. The assessment should be revised if confirmed reports accumulate without public corrections, if the checks fail to detect the behaviors they were meant to cover, or if apparent delivery gains do not translate into useful capabilities. The middleware regression and provenance incident already identify work that needs continued attention.

This is an engineering judgment supported by observable results, with the uncertainty left visible. Time spent, feature mix, architecture, and growing maintainer experience are not controlled here. Sparse public reports cannot establish unchanged reliability, and later releases have had less time to reveal problems.

Check the study

The study covers delivery and public maintenance records from September 1, 2024 through September 3, 2026, with matched historical windows for context. It includes issue and PR screening decisions, sources for individual findings, release chronology, monthly calculations, and all seven experiment outcomes. Unknowns remain in the data.

Read the methods · Download the data · Inspect the case records · Repeat the experiments · Challenge a claim or suggest a correction