Methods and reproducibility

Hypothesis and study design

After agent-authored code first entered Counterfact's main branch, development output would increase without worsening the rate of newly introduced externally reported defects, report-to-release time, or automated test coverage.

The study is a retrospective observational self-audit. The hypothesis was formulated after the observation period and was not preregistered. Results can describe what changed in this repository; they cannot isolate the effect of AI from changes in testing, review, architecture, contributors, feature mix, or maintainer experience.

Intervention and comparison windows

The intervention begins at March 10, 2026 at 16:38:20 UTC, when the first agent-authored product change entered main. The parent of that merge, 68a3f505, is the pre-adoption repository snapshot.

A supplementary screen covers January 1 through September 3 in both years. It preserves all records collected in the original audit and tests whether the primary boundary changes the origin pattern.

Population

The source population is public GitHub issues and accepted standalone defect pull requests opened by human reporters who were not Counterfact organization members or maintainers at the time. Outside contributors remain external. Searches were rerun without relying on labels.

A record enters the product-defect cohort only when it describes confirmed faulty behavior. Maintainer-authored reports, automated bug-hunt issues, enhancements, invalid-input reports, and questions without confirmed faulty behavior are excluded. A pull request tied to a qualifying issue is evidence for that issue, not a second report. Standalone accepted defect pull requests remain eligible.

The complete review population contains nineteen candidates: fourteen included product defects, three exclusions, one deduplicated pull request, and one process-quality incident. Each disposition appears in thecandidate ledger with its rationale and primary source.

Primary outcomes

1. Introduced product defects per 100 non-dependency merges. A defect is introduced during a window when its first confirmed affected public release was published inside that window. The report must also be opened inside the window.
2. Report-to-release time. Calendar days from the external report to the first public release containing the accepted correction, summarized by the median to reduce sensitivity to long-lived outliers.
3. Automated verification. JavaScript and TypeScript files containing explicit it/test declarations, Python files named test_*.py, the number of those JavaScript and TypeScript declarations, and the public Coveralls line-coverage result at the pre-adoption and end-of-observation snapshots.

Pull-request and release counts measure delivery activity, not product value. Dependency-only pull requests are removed from the primary delivery denominator. npm downloads and GitHub stars appear only as exposure context.

Defect-origin classification

1. Same-year regression: supported behavior demonstrably worked before a specific change in the report year and failed afterward.
2. Defect in a same-year feature: the applicable capability first appeared in the report year and was defective in its initial release.
3. Pre-existing: the failure is confirmed before the report year without an intervening working state.

The rules are applied in that order. Every product defect receives one primary category. The post-adoption analysis adds a separate time boundary: a 2026 feature defect can still predate adoption when its first affected release shipped before March 10.

Edge-case policy

• A refactor that exposes an older latent defect does not become its origin unless it changed previously correct observable behavior.
• A missing edge case in a long-standing subsystem is pre-existing when the same valid input would have failed before the comparison boundary.
• A newly introduced capability is a feature defect when it never worked correctly; it is a regression only if an earlier released version worked.
• “Regression test” in a fix description means the test prevents recurrence. It does not date the origin of the bug.

Evidence standard

Each case begins with the report, accepted fix, merge commit, and first release containing the correction. The faulty logic is traced backward using file history, blame, rename-aware history, content searches, tags, and npm publication dates. Historical states are compared around suspected origins and at the reporter’s version where applicable.

Classification confidence and chronology confidence are recorded separately. “Exact origin” is used only when history identifies the introducing change. “Confirmed affected by” is a conservative bound: the linked commit and release demonstrate that the defect existed by that point without claiming it began there. Each case appendix exposes the source history and release chronology used for its classification.

Maturity sensitivity

A supplementary calculation keeps only cases whose first confirmed affected release had at least ninety complete days of observation before its cohort’s September 4 cutoff. All fourteen product cases meet that threshold. This does not make later releases defect-free; it prevents them from supporting claims for which they had too little observation time.

Reproduction and data availability

The machine-readable evidence manifestcontains the hypothesis, exact windows, intervention commit, all nineteen candidate dispositions, all fourteen case histories, the process incident, activity measurements, source URLs, and feature chronology. The repository’s offline verifier checks window membership, totals, deduplication, source completeness, maturity, and pull-request author sums without depending on a live search result.

Limitations and disclosure

• External reports are not the complete defect population. Internal findings and unknown, unreported bugs are absent.
• Fourteen defects are a small sample, and one reporter supplied six of the 2026 issue reports.
• GitHub authorship records provenance, not sole responsibility. Agent-authored work still passed human specification, review, integration, and release decisions.
• Pull requests, releases, tests, coverage, downloads, and stars measure different dimensions and cannot be combined into a universal quality score.
• The package reorganization changed the coverage denominator, so the public percentages do not describe an identical set of files.
• The absence of an observed post-adoption defect introduction does not prove that none exists.

The audit was conducted and written by Counterfact’s maintainer, who adopted and benefits from coding-agent workflows. Coding agents assisted with the repository work. Source links, explicit exclusions, checked calculations, and the complete candidate ledger are provided so readers can challenge the author’s decisions directly.