Define Test Strategy

Objective

Define the test approach (prefer Test Trophy over Pyramid when integration-heavy), choose frameworks per layer, establish coverage targets, plan test data management, and configure CI integration for tests.

Seed only: Record motivators, layer weights, and framework choices for inclusion in docs/architecture/SAO.md § Test Strategy via DTA-18. Do not finalize AT/E2E directory layout or step-library paths here — TFK-01 fleshes those details into the same SAO section before BPE.

Decisions to Make

1. Trophy / Pyramid & Coverage Targets

Define ratio and targets per level:
- Unit tests — Isolated logic, services, utilities. Target: thin / focused coverage.
- Integration tests — Views, DB queries, service interactions. Target: all CRUD paths (main bet when using Trophy).
- Acceptance (AT) — BDD .feature scenarios; executable requirements.
- E2E tests — Full user journeys via browser. Target: critical paths only.
- Contract tests — API contract validation (if applicable).

Define what "100% pass rate" means (all non-@wip tests must pass before declaring feature complete).

2. Frameworks per Layer

Choose frameworks for each test level:
- Unit: pytest, unittest, Jest, Vitest
- Integration: pytest + Django test client, supertest
- AT: behave-django (Django test client in steps)
- E2E: Playwright, Cypress, Selenium
- Contract: Pact, Schemathesis
- Performance: Locust, k6, Artillery

3. Test Data Management

• Fixtures: Static JSON/YAML fixtures loaded before tests
• Factories: Dynamic data generation (factory_boy, Faker)
• DB seeding: Management commands for consistent test state
• Isolation: How are tests isolated? Transaction rollback? Fresh DB?
• Makefile targets: make test, make test-unit, make test-at, make test-e2e

4. CI Integration

• When do tests run? On every push? On PR? Nightly?
• Failure gates: Which test failures block merge/deploy?
• Test reporting: Where are results published?
• Flaky test policy: How are flaky tests handled?

5. Scan Skills

Query Playbook Skills where capability_domain in:
- TEST_UNIT
- TEST_INTEGRATION
- TEST_E2E

Report coverage and gaps.

6. Agent test lanes (when SAO §17 applies)

When the project has in-app LLM agents (SAO §17 is not "Not applicable"), seed four agent test lanes in the Test Strategy section for TFK-01 to flesh out before BPE:

Seed only here: record that lanes 1–2 belong in default PR quality gates when agents are enabled; lanes 3–4 use separate agent-eval.yml (workflow_dispatch + schedule) and do not block merge.

Do not finalize PRF test file paths, tests/support/agent_story.py, tests/fixtures/llm_scripts/, Makefile targets (test-agent-proof, test-agent-quality), or pytest markers here — TFK-01 fleshes lane wiring, directory layout, and CI/Makefile integration into SAO § Test Strategy before BPE.

Deliverables

• ✅ Test approach (Trophy/Pyramid) defined with coverage targets
• ✅ Frameworks chosen per test level
• ✅ Test data management approach defined
• ✅ CI integration rules established
• ✅ Skill coverage assessed for this domain
• ✅ Seed decision recorded for SAO.md § Test Strategy (DTA-18); TFK-01 will flesh the section before BPE
• ✅ Agent test lanes seeded (when SAO §17 applies); TFK-01 will flesh lane details before BPE