hidkit-eval

## What it measures

- **Hidden tests:** a grader that the candidate never sees checks the behavior.
- **Judge claims:** two blind Opus judges grade scope, redundancy, evidence, readability, and the security report.
- **Cost:** the cost ratio of the Cheffy arm to the baseline arm.
- **Noise:** repeats of each scenario. A gain counts only when it is larger than the noise.

Why a separate measurement

Cheffy has an eval recipe, a small blind comparison of variants. The recipe itself calls its result a judgment, not a measurement. A release gate needs more: repeats, hidden tests, and a noise check.

hidkit-eval is that measurement. It lives in a private repository. If its hidden tests and reference patches reached a model training set, the scenarios could not measure anything.

The method

Each scenario runs in two arms: the baseline arm with plain Claude Code, and the Cheffy arm. The snippet lists what each run gets scored on.

• Two sets. An easy set checks for no regression and for cost. A hard set checks quality, with a train split and a frozen test split.
• One patch per round. Each round adds one patch to Cheffy, and the test split stays frozen.
• Rules for a final run. A feasibility check that spends no tokens runs first, on the exact split. The noise statistic is fixed before the run, not after it. Two judges grade, because one judge gives no agreement value.

The graders run code that the candidate wrote, so they run in a macOS sandbox. The sandbox has no network, allows writes only to the work directory, and starts no binary except node and git.

Every run uses my subscription. The runner removes ANTHROPIC_API_KEY, so a run never bills the API.

The honest result

The exit criterion of Phase 1 is quality above plain Claude Code at a cost of 2 times the baseline or less. That criterion is not met yet.

• Cheffy had no regression on any task.
• Its quality gain (0.10) was below the noise (0.20).
• Its cost was 3.6 to 4.5 times the baseline.

So Hidkit stays in Phase 1. The number tells me where to work next: lower the cost, and find a gain that clears the noise. That is why the measurement comes before any release.