Scoring engine — calibration

The scoring engine is deterministic and published, so you can read exactly what it counts. That tells you what the numbers are; it does not tell you whether they are any good. This page describes the harness that asks the second question, what it has found so far, and what is still wrong.

The harness is a development instrument. It is not shipped, not mirrored to the public collector repo, and its judge never enters the scoring path — a score returned by this API is always produced by the deterministic rules in weights.ts.

How the comparison works

Two independent pipelines score the same repositories on the same dimensions:

• Model. The exact product path — the shared collect path, then scoreProfile(). Numbers are identical to what cosq inspect, the dashboard, and this API report. It is not a reimplementation, so the grid cannot be right while the shipped code is wrong.
• Judge. An LLM reads the raw Claude Code, Codex CLI, and Cursor transcripts for the same repositories, rendered to a compact chronological text by a fixed selection rule, and scores each dimension against a rubric. Transcripts pass through the public redactText engine before any byte leaves the machine, and the harness runs on the operator's own projects — never a user's. Each repository is scored three times; the per-dimension median is used.

Shipping Momentum is excluded: the judge sees sessions, not git history, so it cannot assess it fairly.

Where the two disagree, the resolution is to read the underlying session and decide which side is wrong. Neither wins automatically.

Pre-registration

The judge setup, the metrics, and the acceptance gate were committed to the repository before the first comparison ran, following the same convention as the E1 protocol. Thresholds cannot be tuned to fit one rater and then declared validated. Amendments are append-only and dated, and they record the hypotheses that failed alongside the ones that shipped — including two attempts to treat prompt length as evidence of planning, both measured and both reverted.

Metrics per dimension: Spearman rank correlation (primary), mean absolute delta (secondary), saturation rate (share of non-null cells at exactly 0 or 100), and coverage.

What it found

The harness has been far more valuable as a defect finder than as a target to optimise toward. The largest movements came from detection bugs, not threshold changes, and three of the four were the same underlying mistake: machine- authored text counted as the developer's own behaviour.

The judge was reading the same injected scaffolding the collector was. Both sides were fixed, and the renderer now imports the collector's exported predicate rather than reimplementing it, so the two cannot drift on what counts as a human prompt. Comparisons made before that fix are contaminated on both sides and are not comparable to later runs.

Where the judge was overruled

Two repositories scored exactly 0 on Task Framing where the judge read 58 and 40. Reading the raw prompts confirmed the zeros: four unstructured one-line requests in the first case, and in the second a single session whose structure was pasted verbatim from a course assignment. Both judge rationales said so themselves while still awarding points — "the deeper planning structure came from the assistant rather than the developer", "framing came from the source document, not deliberate structuring."

The judge credits structure regardless of who authored it. The same pattern appears on Tool & Workflow Judgment, where it discounts plan-build-verify shape it attributes to the agent. That is a definitional disagreement between the rubric and the published methodology, recorded as a finding rather than tuned away. No rule was widened, because any rule loose enough to reach these cases would credit a pasted template as the developer's own planning.

Known limitations

1. The local corpus is volatile. Claude Code prunes transcripts after about 30 days by default, so the same repository scores differently across a prune boundary with no code change. Two consequences: a baseline is only comparable to a judge run over the same store state, and only a control/treatment pair over identical store state can measure a code change. In the product, a coverage gap between two analyses withholds the week-over-week delta rather than narrating a drop that was an accounting artefact.
2. Retuning a rule has the same effect. Releases that change a scoring rule are listed in SCORING_RULE_CHANGES; a report spanning one withholds deltas and says the method changed.
3. The judge avoids 0 and 100, so absolute deltas overstate model error at the extremes. Rank agreement is the primary metric for that reason.
4. Small n. With seven repositories, correlations are directional evidence, not statistical proof.
5. Asymmetric corpora. The judge reads a fixed number of recent sessions while the model scores the entire local store, which inflates apparent disagreement on high-volume repositories.

Open work

• Task Framing is too easy to max. The plan-reference rule matches keywords (plan, spec, prd, todo) anywhere in a prompt, which credits a developer for talking about plans. It is the dominant input to the dimension. Replacing keyword presence with something structural is the next round.
• Full-credit thresholds below 1.0 make 100 reachable by design. Task Framing awards 100 at an 80% framed rate. Clamping the range was considered and rejected: it would make the saturation gate pass by construction and contradict the published methodology. Raising the bar is the honest fix.
• Saturation gate not passed. The current gate counts only endpoints not corroborated by external evidence; the latest run sits at 4 of 41 cells uncorroborated, and it is not declared passed, because the census moved for reasons unrelated to any fix.

The standing limitation applies unchanged: these are developmental signals, not validated instruments for consequential decisions. This page exists so that the state of that validation is legible rather than implied.