Validate Test Health
TFK-8 Order: #8 Construction Has Dependencies
Updated 4 weeks, 2 days ago
Guidance
Validate Test Health
Objective
Periodic health check of the test suite: analyze coverage quality (not just quantity), flag anti-patterns, identify tests that need strengthening, and verify all test levels pass.
Process
1. Run Coverage Analysis
make test-coverage
Review coverage report, but remember: coverage is meaningless without quality.
2. Quality Audit — Flag Anti-Patterns
Check for typical mistakes:
Useless Tests:
- Testing default Model methods (model.save(), model.delete(), model.__str__())
- Testing Django auto-generated views without custom logic
- Tests that only check return codes without verifying side effects
Testing Implementation, Not Behavior:
- Test checks save_changes() returns OK but never verifies changes actually persisted
- Test is tightly coupled to implementation details
Fat Controller Signal:
- Logic in views/controllers instead of services → tests are hard to write
- Guideline: Services do the thinking → Views package and provide formatted data → Templates render it
Mocking Violations:
- Any mock/patch in integration tests → violation of no-mock rule
3. Verify All Test Levels Pass
make test # unit + integration
make test-at # acceptance
make test-e2e # E2E (if staging available)
When SAO §17 applies:
make test-agent-proof # lane 1 (when AGENTS_ENABLED=true)
make test-agent-live # lane 3 (nightly / manual)
make test-agent-quality # lane 4 (promotion band)
4. Produce Health Summary
Categorize findings as CRITICAL / WARNING / INFO.
Agent eval health audit (when SAO §17 applies)
Skip this subsection when SAO §17 is N/A.
FakeLLM / ScriptedLLM script hygiene
| Check | Severity | Action |
|---|---|---|
FakeLLM or test double returns [] / empty string on queue exhaustion |
CRITICAL | Replace with explicit raise; add reject_exhausted script variant |
| Script turn count < PRF test's expected agent loop iterations | WARNING | Extend llm_scripts/<prf-id>/happy.json |
Live LLM used in @agent_proof test |
CRITICAL | Move to @live_llm or switch to ScriptedLLM |
assert_agent_story beats missing for declared PRF row |
CRITICAL | Add happy/reject test per rule do-assert-agent-story |
Tool / domain mocks in @agent_proof
| Check | Severity | Action |
|---|---|---|
@patch on ToolExecutor, broker, or domain service in @agent_proof |
CRITICAL | Remove patch — lane 1 requires real DB + real tool registry |
| Asserting assistant chat prose instead of trace beats | WARNING | Refactor to assert_agent_story control-plane keys |
| Lane 2 deterministic test uses ScriptedLLM | INFO | Use plain pytest — no LLM for D0/parse/validate |
Carve-out: only CAP-004 ScriptedLLM may stand in for the LLM in integration tests (rule do-not-mock-in-integration-tests ALTER).
PRF coverage vs SAO §17
Build a matrix from SAO §17 PRF checkbox table vs repo:
| SAO §17 PRF row | llm_scripts/ present |
@agent_proof test |
happy + reject | Status |
|---|---|---|---|---|
| e.g. PRF-SC01-04 | yes/no | yes/no | yes/no | PASS / GAP |
CRITICAL gaps: in-scope PRF row with no linked test or missing reject variant when artifact 56 marks adverse.
WARNING gaps: test exists but catalog row in Fixture Library missing sao_ref / linked_test.
TASK coverage vs SAO §17 (lane 4)
| SAO §17 TASK row | agent_tasks/ oracle |
@quality test |
pass_band documented | Status |
|---|---|---|---|---|
| e.g. TASK-SC02-01 | yes/no | yes/no | yes/no | PASS / GAP |
Lane 4 gaps are WARNING until promotion window — do not fail PR CI for missing TASK tests.
Include agent audit findings in the TAF-08 health summary with lane tags (L1–L4) for TFK-09 scoreboard input.
Details
- Order:
- #8
- Phase:
- Predecessor:
- TFK-7 Add Step and/or Fixture to Library
- Successor:
- TFK-9 Prepare Test Report
- Created:
- May 27, 2026
- Last Updated:
- Aug 21, 2026
Workflow
Test Automation Framework
Establish and maintain the test automation infrastructure following the Test Trophy model (integration-heavy, no mocking). Manages the behave+pytest harness, BDD …
View WorkflowAssigned Agent
No agent assigned
Required Skills
No skills linked
Rules
No rules linked.
Input Artifacts 3
-
E2E Test Configuration
Code
Required
Produced by: Establish E2E Infrastructure -
Fixture Library Catalog
Document
Required
Produced by: Build Fixture Library -
Step Library Catalog
Document
Required
Produced by: Build Step Library
Output Artifacts
No output artifacts