collective

Browse / suites / github.com/storytold / study

Case study

What the repeated and abandoned work in github.com/storytold costs, in tokens an agent would spend, and what keeping a fork current costs against it. Figures are estimates: ranges run from low to high assumptions, the mark is the middle one.

Findings

• Adopting beats reimplementing: reviewing an existing pull request, rebasing it onto today's upstream and rebuilding it costs about 28% of the tokens it took to write (range 22% to 41%), and about a tenth when the rebase is clean, as it was for 26% of open PRs. Tests run either way, so they drop out; through Collective they also run sandboxed in the project's Nix environment, which costs compute but adds a safety check. The 1,832 redundant implementations here cost about 3.9B tokens; adopting the first one each time would have cost about 1.1B.
• Of 27,763 pull requests by 2,793 authors, 17,336 merged, 2,566 were closed unmerged and 7,861 are open.
• Merges are fast: half merge within 9 hours and 97% within a week, so a PR still open after a week goes on to merge 18% of the time, and 8% after two weeks.
• Duplicated effort: 1,040 features were implemented by two or more authors (2,872 PRs, 1,832 of them redundant): about 3.9B tokens (range 1.1B to 12.0B), of which 33.8M is the redundant code itself.
• Wasted effort: about 27.2B tokens (range 7.7B to 84.5B) went into PRs that were closed unmerged, superseded by a merged duplicate, or are unlikely ever to merge, including 18,913 rebases of them; the code itself is 172.0M.
• Keeping a change current costs least when it is rebased once, at the end; against that, syncing every 14 days 1.1x, every 7 days 1.5x, every 3 days 2.4x, every day 4.3x, every upstream commit 21.2x as much over a PR's life, because upstream keeps rewriting the same lines and each rewrite is resolved again.
• A fork synced weekly costs about 2.7M tokens a month to keep current; a redundant implementation costs about 2.1M and a wasted PR about 2.8M, so a weekly fork pays for itself if it prevents about 1.3 duplicate implementations a month.
• At weekly syncs, 67% of syncs in these repositories (about 100 upstream commits a day in the busiest) need conflict work; mechanical syncs cost nothing, so the cadence should be set per fork by how often its syncs conflict, starting weekly.
• Caveats: token figures are a model (bytes/4, with stated multipliers), not measured usage; upkeep was simulated on 400 sampled open PRs spanning a median of 8 days, so the cost of forks kept for months is extrapolated; no PR here is older than a month.

Token model

Tokens ~ bytes/4. Work set W = bytes of files a PR modifies at its merge base + its diff. Development = T reads of W + O writes of the added code; each rebase (force push or merge of upstream into the branch) = R reads of W; a conflicted sync also reads and rewrites its conflict lines. Multipliers are assumptions bracketing agentic sessions, not measurements.

{
  "O_dev": [
    1,
    2,
    4
  ],
  "R_sync": [
    1,
    3,
    10
  ],
  "T_dev": [
    5,
    20,
    60
  ],
  "chars_per_token": 4,
  "conflict_line_tokens": 12,
  "max_pass_tokens": 200000
}