OverviewHistorical evidence — not rerun
docs/eval-report.md
# Evaluation report
Ground truth: `seed/eval_labels.json` (48 criteria; see its `_about` for how they were labelled).
Labelled by: **Claude Code (AI); two-pass independent agreement check on a 20-item sample; disagreements adjudicated by a third AI pass, pending human review**.
Positive class for precision and recall: **fail** (unbuilt work marked done is the costly mistake).
## Runs by prompt version
| Run | Prompt | Model requested | Model that answered | n | Accuracy | Fail precision | Fail recall | Unclear rate | Cost | How it ran |
|---:|---|---|---|---:|---:|---:|---:|---:|---:|---|
| 1 | heuristic-v1 | heuristic | heuristic | 48 | 60% | 64% | 56% | 27% | $0.0000 | offline keyword baseline |
| 2 | v1 | claude-sonnet-5-5 | claude-sonnet-5-5 | 48 | 96% | 94% | 94% | 8% | $1.2513 | backend=claude-cli (claude -p --output-format json) |
| 3 | v2 | claude-sonnet-5-5 | claude-sonnet-5-5 | 48 | 100% | 100% | 100% | 8% | $1.2186 | backend=claude-cli (claude -p --output-format json) |
## Latest run (v2, claude-sonnet-5-5)
Confusion matrix (rows = human label, columns = checker verdict):
| label \ verdict | pass | fail | unclear |
|---|---:|---:|---:|
| **pass** | 28 | 0 | 0 |
| **fail** | 0 | 16 | 0 |
| **unclear** | 0 | 0 | 4 |
## What it gets wrong (0 of 48)
| Criterion | Label | Verdict | Reason group | Checker's reasoning |
|---|---|---|---|---|
## Label check: two independent AI passes
Labels are AI-generated with a two-pass independent agreement check. Ed specified the process; a third independent AI pass adjudicated the disagreements, which are listed for human review.
The 20 criteria in the relabel sample were labelled twice from scratch, each pass in its own fresh Claude Code headless session with no tools and no access to the repository, the stored labels or the other pass. A third fresh session adjudicated each disagreement without seeing the stored labels. Model reported: claude-sonnet-5-5. Reported cost: $0.43. Details, prompts and raw responses: `seed/relabel_two_pass.json`.
| Comparison | Agreement | Cohen's kappa |
|---|---:|---:|
| Pass 1 vs pass 2 | 17/20 (85.0%) | 0.72 |
| Adjudicated vs stored labels | 18/20 (90.0%) | 0.82 |
### Disagreements between the passes (3)
| Criterion | Pass 1 | Pass 2 | Adjudicated | Stored | Adjudication note |
|---|---|---|---|---|---|
| AC-CLICK-102-4 | **pass**: Invalid colours raise an error, but the message does not list the valid colour names, so only part of the criterion is met. | **fail**: The error is raised, but the message only echoes the invalid value and does not list the valid colour names. | **fail** | fail | The error is raised, but the message 'Unknown color {colour!r}' only echoes the invalid value and doesn't list the valid colour names, so only part of the criterion is met. |
| AC-CLICK-202-4 | **fail**: The diff adds help for arguments but no locale or translation-catalogue loading for help output. | **unclear**: The commit adds argument help and wraps some strings in _(), but whether the help output is translated at runtime depends on catalogue behaviour the diff cannot show. | **unclear** | fail | The diff adds argument help and wraps new strings in _(), but it shows nothing about locale or catalogue loading. Whether help output is translated at runtime can't be settled from it. |
| AC-HTTPX-202-1 | **pass**: Helpers moved to _client.py as private functions and the tests moved to tests/client/test_headers.py, next to the client tests. | **fail**: The helpers moved to _client.py as private functions. The tests moved to tests/client/test_headers.py, which is not clearly next to the client tests, and the diff is shortened, so the move is only partly shown. | **pass** | pass | The helpers are now private functions in _client.py (_same_origin, _is_https_redirect, plus _port_or_default). The tests were moved out of tests/test_utils.py into tests/client/test_headers.py, which is in the client tests directory. |
### Where the adjudicated label differs from the stored label (2)
| Criterion | Stored | Adjudicated | Stored rationale |
|---|---|---|---|
| AC-HTTPX-101-4 | unclear | pass | The linked commit moves only _config.py's import under TYPE_CHECKING; whether every import path avoids ssl can't be seen in the diff. |
| AC-CLICK-202-4 | fail | unclear | Nothing in the linked change concerns translation or locales. |
### Observations
- AC-CLICK-102-4: pass 1's label (pass) contradicts its own reason ('only part of the criterion is met'). The disagreement is a labelling slip, not a real difference of judgment; the adjudicated label (fail) matches the reason.
- AC-HTTPX-101-4: both passes labelled pass, but both reasons hedge ('other modules may still import it'; 'not shown'). The stored label (unclear) fits those reasons better. Agreement between passes doesn't guarantee a correct label.
- AC-CLICK-202-4: the passes split fail/unclear and the adjudicator chose unclear; the stored label is fail. Whether 'nothing about translation in the diff' means fail or can't-tell is a rule question, which is exactly what a human reviewer should settle.
Caveat: every labeller here is a Claude model, the same family as the judge. This measures how stable the AI labels are, not whether they are right. A person relabelling the sample (`relabel.html`) is still the stronger check.
docs/evidence/verification-status.txt
$ devtrack status --criteria (2026-11-04T12:49Z, SQLite, seed data)
SYS-01 v1 roll-up Failed roll-up of SUB-CLI-01, SUB-HTTP-01
SUB-CLI-01 v1 roll-up Failed roll-up of REQ-CLICK-01, REQ-CLICK-02, REQ-CLICK-03
REQ-CLICK-01 v2 test Failed pallets/click #1049 2247b35ea1
AC-CLICK-101-1 Verified build #1049
AC-CLICK-101-2 Verified build #1049
AC-CLICK-101-3 Failed build #1049
AC-CLICK-103-4 Verified build #1049
REQ-CLICK-02 v1 test Partially verified pallets/click #1049 2247b35ea1
AC-CLICK-102-1 Verified build #1049
AC-CLICK-102-2 Verified build #1049
AC-CLICK-102-3 Verified build #1049
AC-CLICK-102-4 Not verified
AC-CLICK-102-5 Verified build #1049
REQ-CLICK-03 v1 inspection Partially verified pallets/click #1049 2247b35ea1
AC-CLICK-201-1 Verified build #1049
AC-CLICK-201-3 Verified build #1049
AC-CLICK-201-4 Not verified
AC-CLICK-204-2 Verified on older build build #1028
SUB-HTTP-01 v1 roll-up Failed roll-up of REQ-HTTPX-01, REQ-HTTPX-02, REQ-HTTPX-03, REQ-REQ-01, REQ-REQ-02, REQ-REQ-03
REQ-HTTPX-01 v2 test Needs re-verification encode/httpx #1047 b5addb64f0
AC-HTTPX-101-1 Re-verify re-verify since v2
AC-HTTPX-101-2 Re-verify re-verify since v2
AC-HTTPX-101-3 Re-verify re-verify since v2
AC-HTTPX-101-4 Re-verify re-verify since v2
AC-HTTPX-101-5 Re-verify re-verify since v2
AC-HTTPX-101-6 Re-verify re-verify since v2
REQ-HTTPX-02 v1 test Not verified encode/httpx #1047 b5addb64f0
AC-HTTPX-201-1 Not verified
AC-HTTPX-201-2 Not verified
AC-HTTPX-201-3 Not verified
REQ-HTTPX-03 v1 inspection Failed encode/httpx #1047 b5addb64f0
AC-HTTPX-102-3 Failed build #1047
REQ-REQ-01 v1 test Partially verified psf/requests #1051 d38495c906
AC-REQ-101-1 Verified build #1051
AC-REQ-101-2 Verified build #1051
AC-REQ-101-3 Not verified
REQ-REQ-02 v3 analysis Needs re-verification psf/requests #1051 d38495c906
AC-REQ-201-1 Re-verify re-verify since v3
AC-REQ-202-1 Re-verify re-verify since v3
AC-REQ-202-2 Re-verify re-verify since v3
REQ-REQ-03 v1 demonstration Partially verified psf/requests #1051 d38495c906
AC-REQ-203-1 Verified on older build build #1036 (v2.34.0)
AC-REQ-203-3 Not verified
docs/evidence/audit.txt
$ devtrack audit --verify
audit chain intact: 538 rows
$ devtrack audit --key AC-REQ-202-1
16 2026-06-05T16:00:00Z nina-kowalski verification.recorded AC-REQ-202-1 build psf/requests#1046
482 2026-05-14T12:00:00Z change-control reverification.opened AC-REQ-202-1
484 2026-06-20T16:00:00Z change-control verification.invalidated AC-REQ-202-1 build psf/requests#1046
488 2026-06-20T16:00:00Z change-control reverification.opened AC-REQ-202-1
537 2026-06-05T16:00:00Z change-control reverification.closed AC-REQ-202-1 build psf/requests#1046
docs/evidence/data-quality.txt
$ devtrack dq
ok no_orphans:stories.feature_id
ok no_orphans:tasks.story_id
ok no_orphans:acceptance_criteria.story_id
ok no_orphans:reviews.pull_request_id
ok no_orphans:reviews.reviewer_id
ok no_orphans:commits.author_id
ok no_orphans:pull_requests.author_id
ok no_orphans:task_criteria.task_id
ok no_orphans:task_criteria.criterion_id
ok no_orphans:commit_tasks.commit_id
ok no_orphans:commit_tasks.task_id
ok no_orphans:requirement_criteria.requirement_id
ok no_orphans:requirement_criteria.criterion_id
ok no_orphans:pull_request_commits.pull_request_id
ok no_orphans:pull_request_commits.commit_id
ok no_duplicate_links:task_criteria
ok no_duplicate_links:commit_tasks
ok no_duplicate_links:requirement_criteria
ok no_duplicate_links:pull_request_commits
ok no_duplicate_links:criterion_tests
ok required:features.title
ok required:stories.title
ok required:tasks.title
ok required:acceptance_criteria.text
ok required:requirements.body
ok required:commits.message
ok required:pull_requests.title
ok required:people.display_name
ok backlog_tasks_have_story
ok merged_prs_have_merge_time
ok stories_have_criteria
ok no_failed_staging_rows
ok requirement_tree_has_no_cycles
ok verification_on_own_repo_build
ok no_duplicate_source_keys
35/35 checks passed