Hypothesis: After agent-authored code first entered Counterfact’s main branch, development output would increase without worsening the rate of newly introduced externally reported defects, report-to-release time, or automated test coverage.
Did AI-assisted development change Counterfact’s software quality?
Patrick McElhaney via Codex · September 4, 2026
Counterfact is a TypeScript tool that turns an OpenAPI description into a live, stateful mock API. On March 10, 2026, I began merging work authored by coding agents into the project. I wanted to answer a simple question: has quality increased, decreased, or stayed the same since I started using AI?
Abstract
On the measures available, quality increased. The audit found no externally reported product defect whose first affected release entered the post-adoption window. Median report-to-release time fell from six days to one, test files increased from 36 to 72, explicit test declarations from 270 to 904, and reported line coverage from 82.884% to 91.73%, while non-dependency delivery increased from 15 to 281 merged pull requests. These observations are not a causal estimate. They do not establish that AI produced the improvement, and they do not measure unreported defects.
Research question
The concern behind this study is straightforward. Coding agents can produce changes much faster than a maintainer working alone. If review and verification do not keep pace, that additional output could simply convert speed into defects. Counterfact offers a useful, bounded case: its repository and release history are public, its behavior spans code generation and a live HTTP runtime, and external users report concrete failures through GitHub.
The hypothesis therefore has two parts. Output should rise, and three observable quality indicators should not worsen: defects introduced during the study window, time from report to corrected release, and automated verification. A failure on those indicators would count against the hypothesis even if more features shipped.
Methods
This is a retrospective observational self-audit, not a controlled or preregistered experiment. The intervention begins at the first agent-authored product change merged to main on March 10, 2026. The primary comparison uses the same March 10–September 3 interval in 2025. The broader January 1–September 3 cohorts remain as a sensitivity check.
The defect population contains public issues and accepted standalone pull requests opened by external humans and confirmed to describe faulty product behavior. Each included case was traced backward through source and release history. A report filed after adoption does not count as an AI-period defect unless its first affected release also entered during that period. Enhancements, invalid-input reports, duplicates, and a release-provenance incident are recorded but not counted as product defects.
The complete rules, exact timestamps, queries, and evidence standards are in the methods appendix. Every reviewed record is listed in the candidate disposition ledger.
More reports, but none traced to the intervention period
External users reported seven confirmed product defects after adoption, compared with three in the matched 2025 interval. That increase is visible and should not be explained away. Source history changes its interpretation, however: all seven post-adoption reports concerned behavior already present before the first agent-authored change reached main. Six were old edge cases. The seventh belonged to an operation-ID feature first released on February 26, twelve days before the intervention.
The finding is therefore not “fewer bugs.” It is that the observed post-adoption reports did not identify a product defect introduced in the post-adoption release cohort, despite a large increase in delivery. This population cannot reveal defects that users did not encounter or report.
Corrections reached users faster
Median time from report to the first release containing the accepted correction fell from six calendar days in the matched 2025 cohort to one day after adoption. All seven post-adoption reports reached a corrected release within five days. This measures release response, not the effort required to diagnose, review, or test each correction.
The broader audit gives the same origin pattern
The supplementary January 1–September 3 screen contains 4 product defects in 2025 and 10 in 2026. The 2026 set contains 8 defects predating 2026 and 2 defects born with a feature first released in February 2026. It contains no regressions caused by a 2026 change. All fourteen cases meet the 90-day maturity threshold used in the audit, so removing newer releases does not change those totals.
Three 2026 product records fall before the AI-adoption boundary. They remain visible in the defect appendix and the candidate ledger, but they are not used in the primary post-adoption comparison.
One post-adoption process failure remains part of the result
Release 2.16.2 was authentic, but it appeared on npm without the normal Git tag, GitHub release, or visible publishing provenance. An external user reasonably questioned whether it was legitimate. The audit records this as a process-quality incident rather than a product defect because the published code was genuine. Excluding it from product percentages does not remove it from the interpretation.
Discussion
The observed combination is consistent with quality increasing while output increased: no post-adoption first-affected release appears in the external defect cohort, corrections shipped faster, and automated verification became broader and deeper. That is enough to reject the simple claim that greater use of coding agents necessarily reduced Counterfact’s quality during this period.
It is not enough to identify the cause. Testing, explicit architecture, tighter review gates, changes in contributors, the kinds of features attempted, and ordinary learning by the maintainer all changed during the same period. Agent-authored pull requests still depended on human prompts, review, integration, and release decisions. A different project or a less constrained workflow could produce a different result.
Limitations
- The sample contains fourteen product cases, and one reporter supplied six of the 2026 issue reports.
- External reports omit internal findings, silently abandoned use, and unknown defects.
- Pull requests and releases are unequal units of work and risk; their counts provide context, not a controlled exposure measure.
- Coverage increased across a codebase reorganized into multiple packages, so the before and after denominators are not identical.
- The hypothesis was formulated retrospectively and the audit was performed by the project maintainer, who uses and benefits from agentic coding tools.
- Zero observed post-adoption defect introductions is not proof that no such defect exists.
Conclusion
The most defensible directional answer is quality increased on the measures available. Counterfact shipped substantially more non-dependency work, resolved external reports faster, and expanded its automated verification without an observed externally reported defect introduced by a post-adoption release. The study does not show that AI caused those changes. It shows that, in this repository and period, the feared quality decline is not present in the evidence examined.
