Eval scorecard
The checker against 48 labelled criteria. Every run is stored per prompt version and model, so a change is measured before it ships. Precision and recall treat fail as the positive class.
Labels are AI-generated with a two-pass independent agreement check. Ed specified the process; a third independent AI pass adjudicated the disagreements, which are listed for human review. On a 20-criterion sample the two passes agreed 85% of the time (Cohen's kappa 0.72). Every labeller is a Claude model, like the judge, so a high score shows the models agree, not that they're right. A person relabelling with
scripts/relabel.py is the stronger check.| Run | Prompt | Model | n | Accuracy | Fail precision | Fail recall | Unclear rate | Cost | When |
|---|---|---|---|---|---|---|---|---|---|
No eval runs yet. Run devtrack eval. | |||||||||