Article / Supplementary data

Supplementary delivery and product data

Defect counts need context. This appendix records how much work entered the project, how automated verification changed, whether public exposure contracted, and which substantial capabilities shipped during the study. None of these measurements is treated as a standalone quality score.

Matched-window delivery activity

Measure	2025 comparison	2026 post-adoption
Non-dependency merged pull requestsExact matched March 10–September 3 windows.	15	281
Published releasesnpm publication timestamps within the matched windows.	5	21
Conventional test filesImmediately before adoption and at the end of observation.	36	72
Explicit test declarationsJavaScript and TypeScript test declarations in conventional test files.	270	904

Non-dependency merges increased from 15 to 281, or 18.7 times. The complete 2026 matched window contains 558 merges: 277 dependency-bot changes, 197 agent-authored changes, 80 maintainer-authored changes, and four changes by other contributors. The 2025 matched window contains 153 merges: 138 dependency-bot changes, eleven maintainer-authored changes, and four changes by other contributors.

Authorship identifies provenance, not review effort or product value. A pull request can be trivial or extensive, and every accepted agent-authored change remained subject to maintainer review and release decisions.

Automated verification snapshots

Measure	Before adoption	September 3
Conventional test files	36	72
Explicit test declarations	270	904
Reported line coverage	82.884%	91.73%

The file count includes JavaScript and TypeScript files containing at least one explicit it or test declaration, plus Python files named test_*.py. Declaration counts include those JavaScript and TypeScript calls, including modifiers. Parameterized tests can execute more cases than this static count implies. The exact counter is available in the repository and both invocations appear under Methods.

Public line coverage rose by 8.846 percentage points, from 82.884% immediately before adoption to 91.73% at the observation endpoint. During that interval the project was reorganized into multiple packages, so the percentages are real results over changing source sets rather than a controlled measurement over identical files.

Full-window sensitivity measurements

The original January 1–September 3 comparison counted 226 merged pull requests and 9 releases in 2025, versus 625 merged pull requests and 28 releases in 2026. It used the final 2025 release snapshot—35 test files and 240 declarations—rather than the more precise March 10 pre-adoption snapshot used in the article.

Public exposure did not contract

Measure	Earlier observation	2026 observation
npm downloads, January–August	55,391	62,132
GitHub stars	131 at year-end 2025	168 on September 3

Downloads include repeated and automated installations; stars indicate interest rather than active use. They are included only to show that the project’s externally visible footprint did not disappear while defects were being observed.

Major capabilities released in 2026

1.5.0, January 22: An HTTP client inside the REPL
Developers could send requests to the simulated API without leaving the interactive session, alongside delay controls for latency testing.
2.2.0, March 19: An Admin API and agent skill
Counterfact exposed a control surface intended for tooling and coding-agent workflows, while preserving the human-facing REPL.
2.6.0, April 6: Request validation and a fluent request builder
Incoming requests could be checked against the OpenAPI contract, and route() added typed discovery, construction, autocomplete, and execution inside the REPL.
2.7.0–2.8.1, April 10–14: Composable scenarios and startup state
Named scenario functions made state transitions reusable; startup scenarios could seed a useful simulated world automatically when the server began.
2.9.0, April 17: Multiple APIs in one process
ApiRunner isolated generation and runtime state per specification and enabled grouped APIs to share one Counterfact server and REPL.
2.11.0, May 5: First-class API versioning
Version-aware generated types, derived prefixes, and $.minVersion() let handlers be shared safely across versions of an API.
2.12.0–2.14.0, May 16–30: OpenAPI 3.2 and Overlay support
QUERY, querystring parameters, streaming and server-sent events expanded protocol coverage; repeatable overlays made it possible to adapt a source document without editing it.
2.15.0–2.16.2, August 1–19: A shared store and modular public packages
A typed, hot-reload-safe store became available to routes and tooling, while focused OpenAPI, generator, runtime, client, and REPL packages opened Counterfact to embedding and reuse.

Sources

Exact query strings, timestamps, commits, and calculation rules are published in the methods appendix.