sidecrew

sidecrew toward v1.0: an independent analysis of the bar, the levers, and the shape

sidecrew should not ship v1.0 against “R ≤ 1.0 at N = 12”. Your own measurements show the bar fails because of a large fixed planning cost per job. The per-task cost is low. The bar also mixes a token count with a dollar question. So the honest move is to keep the old bar and report it as failed. Then replace it with a dated, pre-registered dollar bar, set before new data comes in, that compares against an Opus-alone baseline that also pays its own fixed cost. Define v1.0 by what the tool guarantees, not by what it saves.

TL;DR

How to read this report

Every claim is tagged as one of three kinds. (i) Measured by the brief: numbers the Phase 11b/14c/14d session reports. (ii) External evidence: documentation or published research. (iii) My reasoning: arithmetic or inference, labelled as such.

Several conclusions depend on repository files I have not seen. They are flagged inline and collected here.

The arithmetic that reframes the cost question

(i) Measured: with retrieval, 177,203 normalised Opus tokens at N = 11 and 225,691 at N = 36. Without retrieval, 229,830 at N = 11 and 254,933 at N = 45. W_upper = 7,794.

(iii) Fit. Model the total planning cost as P(N) = A + b·N, so R(N) = (A/N + b)/W.

What the fit implies:

  1. The fixed term is 80–120 times the per-task term. At N = 12, fixed cost is 87–96 % of planning spend. The ~46k fixed harness explains only a quarter to a third of A. The rest is thinking and orientation that Opus does once per job whatever N is. The “~49 % output (mostly thinking)” in ADR-0090 is therefore mostly a fixed cost, not a per-task one. So the target is “make orientation and deliberation cheap per job”, not “make each task cheaper”.
  2. Retrieval moved cost from the fixed term to the per-task term. It lowered A by ~66k but raised b by ~1.2k. The two arms cross at (221,709 − 155,868)/(1,940 − 738) ≈ 55 tasks. Above that, retrieval costs more. This fits planners that query per task instead of orienting once.
  3. The per-task cost b is well below W, 9–25 % of W_upper. The central economic claim, that Opus plans cheaply per task and local workers do the heavy work, has not been falsified. What fails is the per-job entry fee.
  4. The comparator matters as much as the fit.
    • Against the Opus-direct per-task cost Phase 11b measured, break-even moves to 51–69 tasks. With retrieval: 155,868/(5,018 − 1,940) ≈ 51 and 155,868/(4,217 − 1,940) ≈ 68. Without: 221,709/(5,018 − 738) ≈ 52 and 221,709/(4,217 − 738) ≈ 64.
    • The other way round: if Opus working alone also pays a fixed session cost A_d, the true break-even is (A − A_d)/(W − b). With an illustrative A_d of 46k (the size of sidecrew’s own harness), it is ≈ 19 tasks with retrieval.
    • The bar, as far as I can tell from the brief, charges sidecrew a fixed cost and Opus none. That is a construct error, whatever the score. (Needs W_upper’s derivation to confirm.)
  5. Survival changes the picture a lot. The base arm’s survival was 9/11 and 32/45 (i). Cost per surviving task is 229,830/9 = 25,537 (R ≈ 3.28) and 254,933/32 = 7,967 (R ≈ 1.02). Phase 11b found no defect in Opus’s own output. If the direct comparator delivers every task, per-delivered-task R is the fair number, and even the large job barely breaks even. Whether R already accounts for survival depends on the frozen wording (not provided).
  6. The data can’t support much weight. There are two points per arm, one run each, no variance, and different task mixes at each N. N is also an outcome of the planner, not a controlled variable (36 vs 45 at “N≈40”). A planner that splits work more finely lowers R without delivering more. That is a Goodhart hazard built into “tokens per task”.

(ii) Dollars, not tokens. Anthropic’s current pricing page lists:

Anthropic’s thinking docs say thinking tokens “are billed as output tokens”. On Opus 4.5 and later, “the API keeps previous thinking blocks by default, and they count toward the context window like any other input tokens.” So the “counted twice” in ADR-0090 is real in billing, but the two counts are priced 10–50× apart. A token-count R gives a $25-per-million output token and a $0.50-per-million cache read the same weight.

(iii) An illustration, with the assumptions stated: take the N = 11 retrieval run (177k normalised) at Opus 4.8 prices, with ~43k real output tokens appearing twice.

Output is likely 60–75 % of the dollars but only ~49 % of the tokens. This is an estimate. The real split needs the raw usage fields, which the harness should log.

§9 answers

Q1. Is the v1 bar right, and how should a frozen bar be revised?

Bottom line. No. The bar is not wrong because it is hard. It is wrong because it measures the wrong thing: a token count, against a comparator that seems to pay no fixed cost, per planned rather than delivered task. And it gates v1.0 on cost when v1.0 is, by the usual meaning, a stability promise. Keep the old bar, publish that it failed, and adopt a new, dated bar. The new bar should be justified by these construct flaws alone and tested on fresh data. v1.0 itself should be capability-first.

What 1.0 means (ii). In SemVer, the major version tracks public API compatibility. Its FAQ says: “If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0.” sidecrew is published as 0.2.0 with provenance on npm and the MCP Registry. For outside users, 1.0 will read as “the CLI/MCP interface and its guarantees are stable”, not “we hit a cost ratio”.

(iii) Tying the version number to a research milestone mixes two promises. It also pushes the team to redefine the milestone until it passes. A better split:

How to revise honestly (ii).

Applied to sidecrew (iii):

  1. Report R = 2.07/2.68 at N = 12 as a fail under the frozen rule, in the release notes and in DECISIONS.md.
  2. Write an ADR that names the construct defects without reference to the score:
    • token-count weighting versus billed dollars;
    • a comparator with no fixed session cost (if W_upper’s derivation confirms it);
    • per-planned-task rather than per-delivered-task;
    • tokenizer drift across model versions.
  3. Pre-register the new bar before any new measurement. Suggested form: dollars per delivered (gate-surviving) task, sidecrew versus Opus-alone. Both should run the same task list on the same project, with Opus-alone including its own session overhead and test runs. Measure at stated N values (for example 5, 12, 25, 50) with replicates.
  4. Check severity. The new bar must be one the current design could fail on fresh data: a new night, a new directory, ideally project-b.

A bar redefined on data already seen is a goalpost move, however well argued.

Whether cost should gate v1 at all (iii). The owner’s measure was “fewer tokens, faster, more precise”. “More precise” is where sidecrew has an advantage no competitor can easily copy: a machine-checked iff gate and a guaranteed-intact project. Phase 11b’s gate found no defect in Opus’s output on renames, so that precision advantage over Opus has not yet been shown either. A capability-first v1 is still the one sidecrew can defend today, because its guarantees are about the process, not a comparison.

Q2. The highest-leverage path to a v1 worth shipping, ranked by value per effort

Bottom line. Fix the instrument first; it is nearly free and may change the verdict. Next, run the deterministic-executor experiment; it is cheap and may simplify the product. Then cut the fixed planning cost with thinking effort and machine plans. Then fix the gate’s wall-clock and memory. Leave reader work last. Each step comes with the measurement that would prove or refute it.

  1. Re-score existing transcripts in dollars, by token class, against a fair comparator. Effort: hours. Value: may move the bar by 2×.
    • Log input, cache_creation, cache_read, output and thinking tokens separately. Anthropic’s docs point to usage.output_tokens_details.thinking_tokens.
    • Run the matched Opus-alone arm the brief admits it never ran, on the same 11 and 45 tasks. It also gives the missing wall-clock head-to-head.
    • Proof: dollars per delivered task for both arms, three replicates each.
  2. Deterministic-executor arm for the shapes that already survive. Effort: days. Value: could remove the 7B from those shapes and most of the planning.
    • Rename via the TypeScript language service. Unused imports and lint-shaped flags via ESLint –fix or tsc diagnostics. Unreferenced exports via Knip. Each goes through the unchanged gate.
    • Proof: survival, false-refusal rate and byte-diff versus the 7B arm on the same plans. If deterministic tools match on rename (Phase 11b: the 7B was byte-identical to Opus on 18/19), the 7B’s value lies only in the shapes where it is weakest.
  3. A thinking-effort sweep on the planner. Effort: a day. Value: attacks the output-dominated fixed term directly.
    • Opus 4.8 exposes effort controls (low/high/xhigh/max). Anthropic docs say that on these models “at lower effort settings it may skip thinking entirely on easy inputs.”
    • Anthropic also warns that “Changes to thinking parameters … invalidate message cache breakpoints”. Hold effort constant within a session.
    • Proof: A (fixed tokens and dollars) and survival at each setting, N = 11, three replicates. Pre-register a non-inferiority margin on survival, for example no more than 5 points worse.
  4. Machine-generated plans for predicate-shaped jobs, with Opus reviewing one compact plan in one call instead of orienting from scratch (see §10.2). Proof: A and survival compared with the current planner.
  5. Memory admission control. Effort: small. Value: turns 9-of-11 timeouts into real results.
    • Count the measured suite footprint (~8.2 GB per project-a suite) in the concurrency ceiling.
    • Use jest’s --maxWorkers and workerIdleMemoryLimit. Per the docs, a worker that exceeds the limit after a test “is stopped and restarted”.
    • Make back-off trigger on memory pressure by name.
    • Proof: zero swap-induced timeouts over a full night at the chosen concurrency.
  6. A two-stage gate (Q6). Proof: planted-regression recall of 100 %, plus measured tasks per hour.
  7. A reflective-reference guard for deletion shapes, an integrity fix the brief does not list.
    • “Every test that passed before still passes” is only as strong as the tests. Code reached only through TypeORM entity globs or DI tokens can be deleted with tsc clean and every test green if no test exercises it.
    • Deletion tasks should be refused when the target file matches any runtime glob in configuration, or when its symbol name appears as a string literal or is registered through a decorator.
    • Proof: planted-trap tasks (an untested glob-loaded entity; a string-token provider) must all be refused.
  8. A Sonnet planner arm. Sonnet 5 is $2/$10 versus Opus 4.8’s $5/$25 (2.5× cheaper), and Opus 5.5 is $4/$20. Proof: non-inferior survival at lower dollars per delivered task.
  9. Reader work (Q5) last. It goes after the smallest dollar term.

Options this session had not considered (iii):

Q3. Small jobs: route away, make cheap, or accept a large-jobs tool?

Bottom line. Neither routing on N nor accepting a large-jobs-only tool. Keep sidecrew’s gate on every job and drop its planner for small ones. Opus, or a deterministic tool, edits directly, and sidecrew verifies and fingerprints. That costs zero planning tokens and keeps the one asset nothing else offers. The planner and worker pipeline takes over above an empirically fitted threshold, currently somewhere between ~20 and ~70 tasks depending on the comparator.

Reasoning (iii).

External support (ii). The Minions paper (Narayan et al., ICML 2025) found that a naive local-remote chat protocol cut remote cost 30.4× but kept only 87 % of quality. The small model “struggles to (1) follow the remote model’s multi-step instructions and (2) reason over long contexts”. Its decomposed MinionS protocol recovered 97.9 % of quality at a 5.7× cost reduction. The lesson for small jobs is that orchestration only pays when there are enough decomposable subtasks to spread the remote model’s fixed decomposition cost over.

What would settle it. A matched head-to-head at N = 1, 3, 5 and 12 on real tasks, comparing:

Measure dollars, wall-clock and delivered-correct counts. The routing threshold should be fitted from those curves, not from two points.

Q4. Feature work: is there a sound gate, and should it be in v1?

Bottom line. No gate can prove that new behaviour is correct. The best available is a gate that proves “the code meets a spec the user approved, and the spec is not trivially satisfiable”. The evidence says even that is weaker than it looks. Keep features out of v1. If built later, the spec test should be written by Opus or Sonnet (that is “the smart part”), approved by the user, and strengthened with mutation and held-out checks.

External evidence (ii):

(iii) Interpretation. If tests that humans wrote let through 16–28 % wrong patches on the affected instances, a test a human merely approved is a weaker oracle still. Approval checks what the test says, not what it leaves out.

The smallest honest gate (iii). All five conditions must hold:

  1. The approved test fails on the pre-change code and passes after (fail-to-pass). This rules out tests that were already satisfied.
  2. Every previously passing test still passes (the existing 2a condition).
  3. The new test kills mutants of the new code, reusing workload #1’s machinery including option F’s non-crash kill. This rules out tests that only check existence or types.
  4. A second, independently written held-out test must also pass. A different model writes it, the implementing worker never sees it, and it checks against special-casing.
  5. The diff stays within the files the plan named.

Even then, the release notes must say “meets the approved specification”, not “correct”.

Who writes the spec (iii). Y = 2/20 (i) rules out the 7B as spec author. Using Opus or Sonnet here does not break the vision, which assigns “the smart part” to Opus. It only breaks a metric that counts every Opus token as a cost. Measurement: Y for Opus, Sonnet and the 7B on the same 20 NestJS targets, with the IBM-style held-out overfitting rate as a second outcome.

Q5. Reader and planner models: graphify, the 7B, Haiku, Sonnet

Bottom line. Readers attack the smallest dollar term, so none of them is the priority. If one is tried, pick the cheapest that doesn’t lie:

A Haiku reader is cheap in dollars ($1/$5), but it saves at most a few tenths of a dollar per job and costs you the principle that worker inference never touches the Anthropic API. For the planner, test Sonnet against low-effort Opus; that is the larger lever.

Evidence (ii):

(iii) Dollar ceiling on a paid reader. Reading is ~36 % of normalised tokens, about 64k at N = 11. If Opus read those tokens uncached at $5/M, the most a reader could save is ~$0.32 per job. Haiku reading the same 64k and writing a 5k summary costs ≈ $0.09. The net saving of ≲ $0.25 per job is small next to the ≈ $1 output term. So a paid reader is not worth giving up a non-negotiable. graphify (the owner’s skill; I have no external information on it and treat it as a repo-map-like artifact) is worth testing only for whether it shrinks orientation thinking.

Staleness (iii). A persistent graph is safe if keyed by content hash per file plus a hash of the compiler and lint configuration and of any runtime glob patterns. Answers are served only when every contributing hash matches, the same key discipline ADR-0065 appears to prescribe (ADR-0065 not provided). Mtime-only invalidation is not enough under git checkouts.

Measurement design.

Q6. Wall-clock: can the gate be faster without weakening its guarantee?

Bottom line. Yes, but not by static test selection alone, which is unsound for this codebase. The sound design has two stages. Stage 1 filters each candidate cheaply with affected tests. Stage 2 runs the full suite on batches of stage-1 survivors and bisects on failure. The guarantee, that every final state passed the full suite, is kept by construction, because nothing is accepted without a full-suite pass. My estimate (not measured) is ~3× more tasks per hour.

Why static selection is unsound here (ii):

Design (iii):

Batch size arithmetic (iii). Dorfman group testing gives expected full-suite runs per candidate of 1/k + 1 − (1 − p)^k.

Batching hazards (ii). Najafi, Rigby and Shang (ESEC/FSE 2019), studying batch testing at Ericsson, found batching saves most at low failure rates and that “Flaky tests are exacerbated by batching, as the batch size grows the probability that one or more commits will have a flaky test failure also grows.” Google’s Flake Aware Culprit Finding (Henderson et al., ICST 2023 Industry Track), evaluated on 13,000+ test breakages at Google, exists because naive bisection is “unreliable if the test being used for the bisection is flaky”. Use the ADR-0084 flake diagnosis to re-run regression-only failures before bisecting.

Is “faster” the right promise? (iii) For unattended nights, throughput per night and “results ready by morning” matter more than per-task latency. Promise throughput and zero babysitting, and publish per-task latency honestly.

Measurement.

Q7. What would show the design should change shape rather than be tuned?

Bottom line. Three results would each count as a change of shape:

Reasoning (iii):

On the worker model (ii).

Comparable systems (ii). The Minions line of work is the closest published analogue. Its result is that local-remote savings depend on the remote model decomposing tasks into short, local-sized chunks. That supports sidecrew’s symbol-scoped tasks and argues against a long-context 7B reader.

§10 challenged

1. “Route by size; below ~30–40 tasks hand the job to Opus directly.”

2. “Machine-generated plans for predicate-shaped jobs.”

3. “Keep the gate strict; faster with affected-tests-first plus full-suite confirmation on the combined result.”

4. “Try graphify (free) before any paid reader; measure a Haiku reader only in dollars.”

5. “Do not put feature work in v1; keep ADR-0082 as the post-1.0 path.”

6. “Redefine the Cost bar as a new, dated decision, e.g. R ≤ 1.0 at the median plan size after routing.”

Making the next measurement correct by construction

(ii)/(iii) Each defect found this week maps to a structural fix:

Caveats

This site is open source. Improve this page.