How this case study was built

Study design and working hypothesis

This is a retrospective observational study of public records, authored by Codex and commissioned by Counterfact maintainer Patrick McElhaney. Its working hypothesis is: AI assistance can increase delivery while preserving the maintainer’s ability to verify and repair what ships. The hypothesis was formulated after observing the data and was not preregistered. The study therefore describes evidence consistent or inconsistent with that question; it does not establish that coding agents caused any outcome.

The continuous observation interval is September 1, 2024, inclusive, through September 4, 2026, exclusive, in UTC. Adoption is marked at March 10, 2026, 16:38:20 UTC. Monthly charts split neither March nor its bars; pre/post summaries use the exact timestamp. September 2026 is a partial month of three days. Later retrieval and historical experiments do not add observation-period events.

The pinned mainline observation commit is 77f0bc6fd2c3bc935d2eb2da80190369d91f7084. Issue creation timestamps use the same exclusive endpoint.

How the hypothesis is examined

Three observable checks make the hypothesis inspectable. These are descriptive checks, with no significance threshold or success deadline selected to turn the observed result into a prospective test.

1. Did delivery increase? Non-dependency PRs in the matched 2022–2026 windows. Evidence against: Delivery does not exceed the highest observed earlier matched-window non-dependency PR count. Counts measure merged activity, not effort saved or product value.
2. Did confirmed external defects reach public corrections? All seven confirmed external product-defect reports after adoption. Evidence against: A report in the fixed seven-case population has no traceable public correction by the study boundary. Report-to-publication times describe these cases. The separate 90-day historical follow-up boundary is an observation and censoring rule, not a repair target or MTTR.
3. Do the relevant checks distinguish faulty from corrected behavior? All seven fixed experiment cases. Evidence against: A relevant assertion fails to distinguish the affected and corrected behavior, or remains inconclusive. Focused checks, performed retrospectively; not historical whole-suite results.

The delivery comparison measures change across calendar-matched windows. The repair and verification checks establish whether those specific feedback loops were functioning during the study period; they do not establish improvement over an equivalently measured earlier baseline. The article’s decision to continue the approach is the maintainer’s interpretation of this evidence.

Public records and their limits

The collection uses a paginated GitHub Search query for all-state issues, a paginated REST pull-request traversal, relevant issue discussions and pull-request review/file evidence, npm publication metadata, and pinned first-parent Git history. The public-record census contains 395 issue records, including reports needed to identify carry-in work. The PR traversal contains 1970 records overall; 1080 were opened or merged in the observation interval. Direct mainline changes are screened separately because a repair need not have a linked issue.

Search totals, page counts, terminal empty-page checks, normalized snapshots, queries, retrieval dates, source URLs, and hashes accompany the evidence. These checks establish the retrieved public record, not a guarantee that every defect is publicly reported.

Every retrieved candidate has a disposition and rationale. Initial screening uses record text and associated repairs, not bug labels, bot authorship, or titles alone. Rule-screened exclusions are distinguishable from individually adjudicated repairs. Relevant discussions are preserved for identified candidates; exhaustive manual reading of every discussion is not claimed. Uncertain classifications and source gaps remain visible in the public-record census and dataset.

One defect, several independent attributes

• Identity: duplicate reports and their repairing PRs link to one underlying defect. Similar symptoms alone do not prove duplication. A documented reintroduction after a correction is a separately identified recurrence.
• Discovery: external human, maintainer/internal human, automated/agent, or unknown. A bot author or a maintainer’s PR does not establish who discovered the behavior.
• Release status at discovery: already released, found before public release, or unknown. A pre-release finding may still subsequently escape into a release; publication chronology determines whether it enters the known backlog.
• Origin: demonstrably pre-existing, introduced after adoption, or unresolved. A matching faulty file in an earlier published tag supplies a conservative affected bound; it is not an exact introduction date.
• Chronology: earliest documented report, confirmed affected release, correcting change, and first corrected public release. Issue closure and PR merge are not substitutes for a release reaching users.

Accepted repairs need a concrete faulty behavior. Refactoring, hypothetical future failures, feature requests, and process incidents are not automatically product defects. Confidence, source-backed bounds, exact timestamps, and unresolved dates remain explicit. Date-only observations cannot justify sub-day comparisons.

Delivery, corrections, and known backlog

Monthly delivery counts use merged non-dependency PRs and npm public release timestamps for the counterfact package. The counts do not add separate publications of the later scoped workspace packages. Dependency classification considers the change itself, including manually authored dependency updates. Counts describe activity; they do not measure effort, value, reliability, or active use.

Released-defect reports are separated by documented discovery channel. Pre-release findings and findings with unknown release status are listed separately. Corrections count the first public release containing a repair. The start-of-month backlog includes known, reported, publicly affected defects that had not yet reached a corrected release, including identifiable carry-in reports. It is the known public-record backlog: incomplete historical ascertainment and unresolved release status prevent interpreting it as all outstanding defects.

Historical comparisons

Historical comparisons support the study question. Each intake begins March 10 at 16:38:20 UTC and ends June 6 at midnight UTC in its own year; the matched observation window ends September 4 at midnight. Release-origin events require an exact introduction within intake and an external report within that release’s own ninety-day follow-up. A pre-window affected bound excludes a new-introduction claim; an unresolved origin remains a diagnostic, not a zero event.

Response observations are selected independently by report time. Each report has its own ninety-day deadline; observation ends at the earlier of that deadline and the administrative September 4 boundary. A correction counts as observed only when its public release falls within that follow-up. This is a follow-up and censoring rule for a comparable historical analysis, not a repair target, an MTTR measure, or a measure of restoring a service. The supporting table derives event membership, PR/release denominators, response sample sizes, and censoring from individual records.

Seven fixed historical demonstrations

The experiment population was fixed at external cases #1617, #1618, #1619, #1842, #1933, #1971, and #2075. Each attempt uses the same relevant assertion on affected and corrected historical implementations. Contemporaneous regression tests are preferred, with necessary compatibility changes recorded; audit-created reproducers are labeled. Packaging behavior is tested using published artifacts.

Each case receives one outcome: demonstrated detection, did not distinguish, or inconclusive with a diagnosed reason. Installation trouble and unrelated test failures do not establish detection. Focused success does not show that the historical whole suite passed, nor estimate defects prevented. No repository-wide mutation campaign is part of this study. Commands, immutable refs, provenance, and output accompany every attempt.

Reproduction

The repository’s site/study/2026 directory contains normalized source snapshots, adjudications, and experimental artifacts. The pages and offline verifier use the same derivation functions. npm run test:audit, npm run verify:audit, and npm run build check analytical boundaries, source integrity, stored totals, and site output.

Download data. The data includes the records, classifications, source links, and calculations used by the public pages.

No fitted causal model, combined quality score, defect-per-download rate, or estimate of “defects prevented” is published in the current analysis. Explanations for changes in outcomes are interpretations, supported where possible by dated workflow evidence.