Activities - Test Automation Framework

#	Name	Abbrev.	Guidance	Actions
Inception 1
1	Define Test Architecture	TFK-1	Define Test Architecture Objective Consume DTA-06 "Define Test Strategy" decisions (already seeded into docs/architecture/SAO.md § …	Read the 'TFK-1 Define Test Architecture' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Define Test Architecture ## Objective Consume DTA-06 "Define Test Strategy" decisions (already seeded into `docs/architecture/SAO.md` § Test Strategy) and flesh out that SAO section with the project's Test Trophy model, runner mapping, coverage policy, directory layout, and zero-bug / zero-length-feedback workflow. Authoritative output: update `docs/architecture/SAO.md` § Test Strategy. Do not treat a separate Test Architecture Document as the source of truth. A thin local pointer file (e.g. `docs/architecture/test-architecture.md` linking to SAO §5) is allowed for navigation only. Handoff: DTA-06 / DTA-18 write motivators and broad strokes. TFK-01 (run before BPE) completes directory layout, AT/E2E runner paths, and quality gates so construction can execute against a finished Test Strategy section. --- ## Decisions to Flesh Out in SAO.md ### 1. Test Trophy (not Pyramid) Our model is the Test Trophy — integration tests are the main bet: \| Level \| Weight \| Runner \| What it proves \| \|-------\|--------\|--------\|----------------\| \| Unit \| Thin \| pytest \| Custom queryset logic, pure utility functions, business logic methods — only where isolation adds value. Do NOT test default Model CRUD, auto-generated views, or framework behavior. \| \| Integration \| Thick \| pytest \| Feature works end-to-end with real DB, real services, real connections. No mocking. Think of them as acceptance tests minus the GUI. \| \| Acceptance (AT) \| Medium \| behave-django \| `.feature` scenarios in `docs/features/` executed against the app via Django test client. Steps from the TFK Step Library. BDD-grade executable requirements. \| \| E2E \| Selective \| behave + Playwright \| Full user journey in `tests/e2e/`. Sequential, slow, runs on staging before release. Screenshots for visual verification. \| ### 2. Why Test Trophy over Pyramid - Unit tests are limited: testing `model.save()` returns OK tells us nothing about whether data actually persisted - Acceptance tests with a real browser are expensive: slow, flaky - Integration tests hit the sweet spot: real thing, no GUI overhead, fast enough for CI - AT tests are integration tests with BDD syntax — the `.feature` file IS the requirements document made executable ### 3. Runner Mapping - pytest: everything up to and including integration tests - behave-django (`make test-at`): acceptance tests — `docs/features/` (spec + runner; Django test client; `tags = ~@wip`) - behave + Playwright (`make test-e2e`): E2E tests — `tests/e2e/` (LiveServer + browser; own `environment.py`) ### 4. Coverage Policy - Coverage metrics are necessary but not sufficient - Coverage is meaningless without quality — 90% coverage with useless tests is worse than 60% with real tests - Quality gates: no mocking in integration tests, no testing framework behavior, no tests > 10 lines without extraction - `@wip` scenarios excluded from CI; all other `docs/features/` scenarios must pass AT before merge ### 5. Zero-Bug and Zero-Length Feedback Document in SAO: bug = failing integration/AT scenario; fix same day with a permanent regression test; continuous pytest feedback via `tests.log`. ### 6. Agent test lanes When SAO §17 (AI Agent Architecture) applies — i.e. the project has agent workflows, tool registries, or LLM orchestration — extend the Test Strategy with a four-lane agent eval model. When SAO §17 is N/A (no agents), skip this subsection entirely; lanes 3–4 and agent Makefile targets are optional no-ops (see TFK-05). Four lanes: \| Lane \| Name \| Runner \| What it proves \| \|------\|------\|--------\|----------------\| \| 1 \| Control-plane proofs \| pytest `@agent_proof` + CAP-004 ScriptedLLM \| Orchestration: workflow id, tools offered/called, domain writes, plan/blackboard transitions — not model quality \| \| 2 \| Deterministic shell \| pytest integration (no LLM) \| D0 prefilter, JSON parse, validate-before-LLM, schema bounds — pure code paths \| \| 3 \| Live contract eval \| pytest `@live_llm` (nightly / optional CD) \| Provider contracts at temp=0: response shape, tool-call envelope, rate-limit handling \| \| 4 \| Task performance quality \| pytest `@quality` (promotion band) \| Frozen TASK-* golden tasks: structured oracle (recall, graph diff, schema bounds) \| Gate matrix (CI stage vs lane): \| Lane \| PR merge (`ci.yml`) \| Nightly (`agent-eval.yml`) \| Promotion / release \| \|------\|---------------------\|----------------------------\|---------------------\| \| 1 \| ✅ when `AGENTS_ENABLED=true` — `make test-agent-proof` \| — \| — \| \| 2 \| ✅ normal integration job (`make test-integration`) \| — \| — \| \| 3 \| ❌ not a PR gate \| ✅ scheduled / `workflow_dispatch` \| Optional CD cert when `AGENT_EVAL_ENABLED=true` \| \| 4 \| ❌ not a PR gate \| Optional regression run \| ✅ vs last certified pass band before promote \| PRF vs TASK (artifact 56): \| Artifact row \| Lane \| Assert helper \| Fixture root \| \|--------------\|------\|---------------\|--------------\| \| PRF-* (integration proof) \| 1 (+ lane 2 for non-LLM steps) \| `assert_agent_story(trace, …)` \| `tests/fixtures/llm_scripts/<prf-id>/` \| \| TASK-* (golden task) \| 4 \| structured oracle + optional domain asserts \| `tests/fixtures/agent_tasks/<task-id>/` \| PRF rows prove control plane under scripted LLM turns; TASK rows prove outcome quality under live (or promoted) models. Do not use lane 4 to prove 429 retry or HITL gates — those belong in PRF lane 1. Skip when SAO §17 N/A: If the project has no agent architecture section, do not add lane Makefile targets to required CI, do not block BPE/MIN on `agent_proof_command`, and do not create empty `llm_scripts/` / `agent_tasks/` trees. Artifact 11–13 templates add capability, not mandate. Document in SAO § Test Strategy: which PRF-* and TASK-* rows are in scope for this release, lane assignment, and `AGENTS_ENABLED` / `AGENT_EVAL_ENABLED` env conventions. --- ## Deliverables - ✅ SAO.md § Test Strategy fleshed out (Trophy, runners, paths, coverage, zero-bug, zero-length feedback) - ✅ SAO.md § Test Strategy § Agent lanes (when §17 applies): four-lane table, gate matrix, PRF vs TASK mapping, skip rules - ✅ Ready for TFK-02 Bootstrap Test Harness and BPE
Elaboration 5
2	Bootstrap Test Harness	TFK-2	Bootstrap Test Harness Objective Set up the complete test infrastructure: directory structure, pytest configuration, behave-django …	Read the 'TFK-2 Bootstrap Test Harness' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Bootstrap Test Harness ## Objective Set up the complete test infrastructure: directory structure, pytest configuration, behave-django configuration, Playwright integration, Makefile targets, continuous test runner, and the Log Story helper (`tests/support/log_story.py`) used by BPE/MIN caplog gates. Layout rule: AT scenarios live in `docs/features/` (spec + runner — single source of truth). There is no separate `tests/acceptance/` or `features/at/` tree. E2E is a standalone suite under `tests/e2e/` with its own `environment.py` and Playwright-backed steps. --- ## Process ### 1. Create Test Directory Structure ``` tests/ ├── unit/ # pytest -m unit │ └── conftest.py ├── integration/ # pytest -m integration │ └── conftest.py ├── integration/agent/ # pytest -m agent_proof (lane 1) │ └── test_prf_.py ├── e2e/ # behave + Playwright (E2E) │ ├── .feature # journey scenarios │ ├── steps/ # Playwright-backed steps (same Gherkin phrases as AT) │ └── environment.py # LiveServer + browser + screenshots ├── infra/ # CDK assertion tests ├── fixtures/ # Shared fixture data │ ├── seed.json │ ├── presets/ │ ├── llm_scripts/ # Ordered ScriptedLLM queues per PRF-* (lane 1) │ │ └── <prf-id>/ │ │ ├── happy.json │ │ └── reject_.json │ └── agent_tasks/ # Frozen TASK- corpora + oracle YAML (lane 4) │ └── <task-id>/ │ ├── input/ │ ├── expected.yaml │ └── meta.yaml ├── support/ # Shared pytest helpers │ ├── __init__.py │ ├── log_story.py # assert_log_story(caplog, where=..., beats=...) │ └── agent_story.py # assert_agent_story(trace, workflow=..., beats=...) └── conftest.py docs/features/ # BDD spec + AT runner (behave.ini paths = docs/features) ├── act-/ # .feature files per act ├── steps/ # AT Step Library (Django test client) ├── support/ # page registry └── environment.py # behave-django AT lifecycle (atomic rollback) ``` Document the log-story helper import path in root `tests/conftest.py` so BPE/MIN agents discover it. Recipes: skill Pytest Log Story Assertions* (`LOG_STORY_TESTING`). Log-story tests run inside existing `make test` / pytest — no new CI job. When SAO §17 applies, also document `assert_agent_story` import path and the `scripted_llm` / `agent_harness` fixtures (skill Agent Integration Proof Patterns). Agent-proof tests use the existing pytest runner gated by `make test-agent-proof` — no change to default `ci.yml` until PRF tests exist. ### Helper contract ```python def assert_log_story( caplog, , where: str, beats: dict[str, list[str]], level: str = "INFO", ) -> None: """Fail with which beat/key was missing; do not swallow AssertionError.""" ``` ### 2. Configure pytest ```ini [pytest] DJANGO_SETTINGS_MODULE = {project}.settings python_files = test_.py python_classes = Test* python_functions = test_* testpaths = tests addopts = -v --strict-markers --tb=short --log-file=tests.log --log-file-level=INFO markers = unit: Unit tests (fast, isolated) integration: Integration tests (real deps, no mocking) acceptance: Acceptance tests (behave BDD) e2e: End-to-end tests (Playwright, excluded from default runs) slow: Slow running tests agent_proof: Lane 1 control-plane proofs (ScriptedLLM, real DB/tools) live_llm: Lane 3 live provider contract eval (excluded from default PR run) quality: Lane 4 TASK-* golden quality eval (promotion band, not PR gate) ``` Default `make test` / `pytest` collection MUST exclude `live_llm` and `quality` (and `agent_proof` until `AGENTS_ENABLED=true` — see TFK-05). Do not add `-m agent_proof` to bare PR CI until at least one PRF test exists (empty marker selection fails). ### 3. Configure behave ```ini # behave.ini [behave] paths = docs/features format = pretty logging_level = INFO tags = ~@wip ``` Create `docs/features/environment.py` (AT: Django test client / atomic fixtures) and `tests/e2e/environment.py` (E2E: Playwright + LiveServer) separately — engines differ; do not share one environment for both. Makefile: - `make test-at` → `manage.py behave --simple docs/features/` - `make test-e2e` → `manage.py behave tests/e2e/` - `make test-agent-proof` → gated `pytest -m agent_proof` when `AGENTS_ENABLED=true` (TFK-05) ### 4. Configure Playwright Integration For E2E tests using behave + Playwright with LiveServerTestCase under `tests/e2e/`. ### 5. Agent story harness When SAO §17 applies, ship `tests/support/agent_story.py` alongside `log_story.py`. Rule: `do-assert-agent-story`. Skill: Agent Integration Proof Patterns. Helper contract: ```python def assert_agent_story( trace, , workflow: str, beats: dict[str, list[str]], ) -> None: """Fail naming missing beat; trace from agent run recorder (not caplog).""" ``` Fixtures (root `tests/conftest.py` or `tests/integration/agent/conftest.py`):* ```python @pytest.fixture def scripted_llm(prf_id): """Load ordered queue from tests/fixtures/llm_scripts/<prf-id>/happy.json.""" return ScriptedLLM(load_script(f"tests/fixtures/llm_scripts/{prf_id}/")) @pytest.fixture def agent_harness(db): """Real DB, real tool registry, CAP-004 injectable LLM — no domain mocks.""" ... ``` Test naming: `test_<prf>_agent_story_happy` / `test_<prf>_agent_story_reject` when the PRF row is adverse or has a reject sketch. Markers: \| Marker \| Lane \| LLM \| CI \| \|--------\|------\|-----\|-----\| \| `agent_proof` \| 1 \| ScriptedLLM only \| PR when `AGENTS_ENABLED=true` \| \| `live_llm` \| 3 \| Live provider, temp=0 \| Nightly `agent-eval.yml` \| \| `quality` \| 4 \| Live provider, temp=0 \| Promotion band \| Agent-story tests prove control-plane trace beats; log-story tests prove logging beats — complementary, both green in the same commit when declared (see rule `do-assert-log-story`). ## Rules Before bootstrapping the harness, read each Rule below in this playbook (by slug), then apply it. Do not rely on memory of the rule text. Required: - `pytest` - `do-continuous-testing` - `do-assert-log-story` - `do-informative-logging` - `do-test-fixture-data-management` When SAO §17 applies, also required: - `do-assert-agent-story` Activity-specific (not a substitute for the rules above): - Ship `tests/support/log_story.py` (`assert_log_story`) as part of harness bootstrap; log-story tests use the existing pytest runner (no new CI job). - Ship `tests/support/agent_story.py` (`assert_agent_story`) when §17 applies; agent-proof tests use gated `make test-agent-proof` (no change to default PR job until tests exist). ## Success Criteria - Directory layout matches Process §1 (including `tests/support/log_story.py`) - pytest and behave configured - At least one golden `_log_story_` test can import and use `assert_log_story` - When SAO §17 applies: directory layout includes `agent_story.py`, `llm_scripts/`, `agent_tasks/`; pytest markers `agent_proof`, `live_llm`, `quality` registered; at least one golden `test_*_agent_story_happy` imports `assert_agent_story` and runs green under ScriptedLLM
3	Build Step Library	TFK-3	Build Step Library Objective Create a reusable library of Gherkin step definitions that ESM can …	Read the 'TFK-3 Build Step Library' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Build Step Library ## Objective Create a reusable library of Gherkin step definitions that ESM can compose scenarios from when the library already exists. The library grows over time — initial seed covers generic patterns, project-specific steps are added via TFK-07 as needed. Same phrases, different engines: AT steps in `docs/features/steps/` use the Django test client. E2E counterparts in `tests/e2e/steps/` use Playwright. Gherkin vocabulary is shared via the Step Library Catalog; implementations are not merged into one module. --- ## Process ### 1. Organize Steps by Domain ``` docs/features/steps/ # AT — Django test client ├── navigation_steps.py ├── form_steps.py ├── table_steps.py ├── auth_steps.py ├── assertion_steps.py ├── dialog_steps.py ├── common_steps.py └── __init__.py tests/e2e/steps/ # E2E — Playwright-backed counterparts ``` ### 2. Generic Step Patterns Navigation: - `Given {user} is on the "{page_name}" page` - `When {user} navigates to "{url}"` - `Then {user} should see the "{page_name}" page` Forms: - `When {user} enters "{value}" into "{field}"` - `When {user} selects "{option}" from "{dropdown}"` - `When {user} clicks "{button}"` - `When {user} submits the form` Tables: - `Then {user} sees table "{table_name}" with {n} rows` - `Then the table "{table_name}" should contain "{text}"` - `When {user} sorts table "{table_name}" by "{column}"` Auth: - `Given {user} is logged in as "{role}"` - `Given {user} is not authenticated` Assertions: - `Then {user} should see "{text}"` - `Then {user} should not see "{text}"` - `Then the element "{test_id}" should be visible` ### 3. Step Implementation Conventions - All steps use `data-testid` selectors for element targeting - Parameterize generically — avoid hardcoding entity names - Log actions at INFO level - Add module-level docstring listing all steps --- ## Agent proofs vs AT > Footnote — do not add PRF/TASK scenarios to the Gherkin Step Library. PRF-* integration proofs (lane 1) and TASK-* golden evals (lane 4) are pytest integration tests, not behave AT scenarios. They live under `tests/integration/agent/` and use `assert_agent_story` / structured oracles — not `docs/features/steps/`. \| Concern \| AT Step Library \| Agent proofs (PRF) \| Task eval (TASK) \| \|---------\|-----------------\|--------------------\|------------------\| \| Runner \| behave-django \| pytest `@agent_proof` \| pytest `@quality` \| \| Location \| `docs/features/steps/` \| `tests/integration/agent/` \| `tests/integration/agent/` or `tests/quality/` \| \| Assert \| Gherkin Then steps \| `assert_agent_story(trace, …)` \| Golden oracle YAML/JSON \| \| LLM \| None \| CAP-004 ScriptedLLM only \| Live provider (lane 4) \| \| Catalog \| Step Library Catalog \| SAO §17 PRF row map \| Fixture Library `agent_tasks/` \| When BPE or MIN declares `agent_proof_command` or `checkpoint.agent_proof_command`, implement pytest tests — do not invent Gherkin phrases for tool-call traces or blackboard state. Reuse AT steps only for human-facing GUI journeys; agent control-plane behavior belongs in lane 1/2 pytest per artifact 56 Part 4.5. TFK-07 may still add fixtures (`llm_scripts/`, `agent_tasks/`) when a PRF script or TASK golden is missing — that is fixture catalog work, not step library work.
4	Build Fixture Library	TFK-4	Build Fixture Library Objective Create a shared library of test fixtures at multiple scopes (session, …	Read the 'TFK-4 Build Fixture Library' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Build Fixture Library ## Objective Create a shared library of test fixtures at multiple scopes (session, suite, test) using both JSON fixtures and FactoryBoy. Establish patterns for context passing, AutoFixture support, and fixture presets. --- ## Process ### 1. Fixture Organization ``` tests/fixtures/ ├── seed.json # Base data loaded once per session ├── presets/ │ ├── team_preset.json # A group with set of users with specific roles │ ├── project_preset.json # Project with workflows, activities, artifacts │ └── empty_preset.json # Clean slate for isolation tests ├── factories/ │ ├── user_factory.py # FactoryBoy user factory │ ├── model_factories.py # Domain model factories │ └── __init__.py ├── autofixture/ │ └── fixture_creator.py # AutoFixture helper ├── llm_scripts/ # Lane 1 — ScriptedLLM queues (when SAO §17 applies) │ └── <prf-id>/ │ ├── happy.json │ └── reject_.json └── agent_tasks/ # Lane 4 — frozen TASK- corpora (when SAO §17 applies) └── <task-id>/ ├── input/ ├── expected.yaml └── meta.yaml ``` ### 2. When to Use JSON vs FactoryBoy \| Use JSON fixtures when... \| Use FactoryBoy when... \| \|---------------------------\|------------------------\| \| Data is static reference data \| Data needs to be parameterized per test \| \| Loaded once per session (seed data) \| Created dynamically with variations \| \| Represents "the world as it is" \| Represents "arrange" step of a specific test \| \| Shared across many tests unchanged \| Tests need unique data to avoid collisions \| ### 3. Fixture Scopes - Session-level (`seed.json`): loaded once via `loaddata` in `before_all` - Suite-level (`presets/`): loaded for a specific group of related tests - Test-level (FactoryBoy): created per test function, rolled back after ### 4. AutoFixture Support For lazy fixture creation — create model instances with sensible random defaults when you don't care about specific field values. --- ## Idempotent behave Given Steps (Background + scenario) When a feature file uses `Background:` to seed reference entities and scenarios add their own `Given` steps for the same natural keys (slug, code, username, composite unique): 1. Step definitions MUST use `get_or_create` or update-in-place — never blind `Model.objects.create()` for keys Background may already insert. 2. Document in the Step Library Catalog which Givens are idempotent vs create-only. 3. On `IntegrityError` during AT, first suspect double-seeding — fix the step, not the scenario order. Reference: Skill 45 Pattern 5 (Behave-Django BDD Runner). --- ## Fixture Library Catalog — agent eval entries When SAO §17 applies, extend the Fixture Library Catalog with two agent-specific roots. Skip catalog rows when §17 is N/A. ### `llm_scripts/<prf-id>/` (lane 1 — PRF integration proof) \| Catalog field \| Purpose \| Example \| \|---------------\|---------\|---------\| \| `prf_id` \| Artifact 56 row id \| `PRF-SC01-04` \| \| `path` \| Relative to `tests/fixtures/` \| `llm_scripts/PRF-SC01-04/happy.json` \| \| `variant` \| happy / reject_* \| `reject_429` \| \| `turn_count` \| Expected ScriptedLLM queue length \| `3` \| \| `linked_test` \| pytest module \| `tests/integration/agent/test_prf_sc01_04.py` \| \| `sao_ref` \| SAO §17 checkbox \| `PRF-SC01-04 dual-tool mode` \| File shape: ordered JSON array of LLM response strings or objects consumed by CAP-004 ScriptedLLM. Reject variants must exhaust the queue with an explicit error — never silent empty fallback (see TFK-08). ### `agent_tasks/<task-id>/` (lane 4 — TASK golden eval) \| Catalog field \| Purpose \| Example \| \|---------------\|---------\|---------\| \| `task_id` \| Artifact 56 Part 4.6 row \| `TASK-SC02-01` \| \| `path` \| Relative to `tests/fixtures/` \| `agent_tasks/TASK-SC02-01/` \| \| `sc_xx` \| Scenario family \| `SC-02` \| \| `fixture_ref` \| Frozen input corpus subpath \| `input/corpus.json` \| \| `oracle_type` \| Assert strategy \| `set_recall` / `graph_diff` / `schema_bounds` \| \| `pass_band` \| Promotion thresholds \| `min_recall: 0.95`, `max_extras: 1` \| \| `n_runs` \| Stability runs at temp=0 \| `3` \| \| `linked_test` \| pytest module \| `tests/integration/agent/test_task_sc02_01.py` \| Oracle schema (`expected.yaml` / `meta.yaml`): \| Field \| Required \| Description \| \|-------\|----------\|-------------\| \| `task_id` \| yes \| Matches catalog row \| \| `oracle_type` \| yes \| Drives assert helper selection \| \| `expected_names` \| set_recall \| Ground-truth name set \| \| `expected_graph` \| graph_diff \| Nodes/edges or ops diff spec \| \| `schema_bounds` \| schema_bounds \| JSON schema or field bounds \| \| `pass_band` \| yes \| min_recall, max_extras, max_latency_ms, etc. \| \| `n_runs` \| promotion \| Runs before certifying band \| \| `provider_pin` \| optional \| Model id for scoreboard (lane 3/4) \| Document dependencies (seed preset, factory hooks) in catalog notes. TFK-07 adds new rows when BPE/MIN introduces a PRF script or TASK golden not yet cataloged.
5	Wire CICD Integration	TFK-5	Wire CICD Integration Objective Configure the CI/CD pipeline to run tests at appropriate stages: unit+integration+AT …	Read the 'TFK-5 Wire CICD Integration' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Wire CICD Integration ## Objective Configure the CI/CD pipeline to run tests at appropriate stages: unit+integration+AT on CI (every push), E2E on staging (before release swap). Define pass/fail gates at each stage. --- ## Process ### 1. CI Pipeline Test Stages Modify `.github/workflows/ci.yml` (DCD-04 artifact) to add acceptance test stage: ```yaml jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install dependencies run: make provision - name: Lint run: make lint - name: Unit tests run: make test-unit - name: Integration tests run: make test-integration - name: Acceptance tests # NEW — TAF addition run: make test-at ``` Gate: All three test levels must pass before container build proceeds. ### 2. CD Pipeline E2E Stage Modify `.github/workflows/cd.yml` (DCD-05 artifact) to add E2E stage on staging: ```yaml e2e-staging: needs: deploy-idle runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install Playwright browsers run: playwright install chromium - name: Run E2E against staging run: make test-e2e BASE_URL=${{ vars.STAGING_URL }} DB_URI=${{ vars.STAGING_DB_URI }} - name: Upload screenshots if: always() uses: actions/upload-artifact@v4 with: name: e2e-screenshots path: tests/e2e/screenshots/ ``` Gate: E2E must pass on staging → create release → swap with prod. ### 3. Screenshot Archival - E2E tests capture a screenshot after every step - Screenshots uploaded as GitHub Actions artifacts for human review --- ## Agent eval CI (optional) When SAO §17 applies, add agent eval lanes without changing the default PR job shape in `ci.yml`. Agent-less projects: skip this section; Makefile targets no-op. ### Default `ci.yml` — UNCHANGED for agent-less and baseline PR flow The existing `test` job (unit → integration → AT) remains the merge gate for all projects. Do not add `-m live_llm`, `-m quality`, or unconditional `-m agent_proof` to the default PR workflow. Optional PR extension (agents only): append a conditional step when the repo enables agents: ```yaml - name: Agent control-plane proofs (lane 1) if: vars.AGENTS_ENABLED == 'true' env: AGENTS_ENABLED: true run: make test-agent-proof ``` Lane 2 (deterministic shell) already runs inside `make test-integration`. No separate job. ### Separate `.github/workflows/agent-eval.yml` Lanes 3–4 run outside the PR merge path: ```yaml name: Agent Eval on: workflow_dispatch: schedule: - cron: '0 6 * * ' # nightly lane 3 jobs: live-contract: if: vars.AGENT_EVAL_ENABLED == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: make provision - name: Lane 3 — live contract eval env: AGENTS_ENABLED: true LIVE_LLM_API_KEY: ${{ secrets.LIVE_LLM_API_KEY }} run: make test-agent-live task-quality: if: vars.AGENT_EVAL_ENABLED == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: make provision - name: Lane 4 — TASK golden quality env: AGENTS_ENABLED: true LIVE_LLM_API_KEY: ${{ secrets.LIVE_LLM_API_KEY }} run: make test-agent-quality ``` CD optional cert (lane 3):* in `cd.yml`, add staging agent contract step guarded by `if: vars.AGENT_EVAL_ENABLED == 'true'` — not required for promote when agents N/A. ### Gated Makefile targets (artifact 11) ```makefile test-agent-proof: ifeq ($(AGENTS_ENABLED),true) uv run pytest -m agent_proof tests/ else @echo "skip: agents N/A (SAO §17)" endif test-agent-live: ifeq ($(AGENTS_ENABLED),true) uv run pytest -m live_llm tests/ else @echo "skip: agents N/A (SAO §17)" endif test-agent-quality: ifeq ($(AGENTS_ENABLED),true) uv run pytest -m quality tests/ else @echo "skip: agents N/A (SAO §17)" endif ``` Gate summary: \| Target \| Lane \| PR (`ci.yml`) \| `agent-eval.yml` \| Promotion \| \|--------\|------\|---------------\|------------------\|-----------\| \| `test-agent-proof` \| 1 \| optional if `AGENTS_ENABLED` \| — \| — \| \| `test-integration` \| 2 \| always \| — \| — \| \| `test-agent-live` \| 3 \| never \| nightly / dispatch \| optional CD \| \| `test-agent-quality` \| 4 \| never \| manual / pre-promote \| pass band vs last cert \| Do not block merge on lane 3/4 flake. Empty `agent_proof` collection must not fail PR CI — enable the conditional step only after at least one PRF test lands.
6	Establish E2E Infrastructure	TFK-6	Establish E2E Infrastructure Objective Set up the end-to-end test infrastructure that enables "day in life" …	Read the 'TFK-6 Establish E2E Infrastructure' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Establish E2E Infrastructure ## Objective Set up the end-to-end test infrastructure that enables "day in life" user journey testing via Playwright-driven browser automation. Location: `tests/e2e/` — standalone suite with its own `environment.py`, `steps/`, and `.feature` files. Not nested under AT. AT remains in `docs/features/` with Django test client steps. --- ## Sequencing contract (why order 6 after #189–#191 is correct) Chosen resolution (Galdr option c):* Activities #189–#191 cover AT concerns only. Activity #192 stands up independent E2E infrastructure. No resequence. \| Activity \| TFK label \| Scope \| Needs Playwright / `tests/e2e/` harness? \| \|----------\|-----------\|-------\|------------------------------------------\| \| #188 \| TFK-02 Bootstrap Test Harness \| Creates directory layout for both `docs/features/` (AT) and `tests/e2e/` (E2E placeholders), pytest, `behave.ini` \| No — only scaffolds paths \| \| #189 \| TFK-03 Build Step Library \| AT step implementations in `docs/features/steps/` + Step Library Catalog (phrases) \| No \| \| #190 \| TFK-04 Build Fixture Library \| Shared fixture data / presets used by AT (and later by E2E data load) \| No \| \| #191 \| TFK-05 Wire CICD Integration \| CI gates for unit/integration/`make test-at` \| No (E2E stays staging/release-only) \| \| #192 \| TFK-06 Establish E2E Infrastructure \| Playwright + LiveServer + `tests/e2e/environment.py` + journey features \| Yes — this activity \| \| #193 \| TFK-07 Add Step and/or Fixture \| Extends catalog; may add AT and/or E2E implementations \| As needed \| How #189–#191 exist without E2E infrastructure: AT runs via Django test client against `docs/features/` (`manage.py behave --simple`). That path never starts a browser. Fixtures and CI wiring for AT do not call Playwright. Therefore predecessors #189–#191 are valid without #192 being complete. What #188 already did vs what #192 does: #188 may create empty `tests/e2e/` directories and Makefile stubs. #192 makes that suite runnable (browser lifecycle, LiveServer, screenshots, `--base-url` / `--db-uri`). Empty dirs ≠ working E2E harness. Dual-path step maintenance: Catalog phrases are owned by #189 (TFK-03). AT Python lives in `docs/features/steps/`. E2E Playwright counterparts live in `tests/e2e/steps/` and are added when journeys need them (#192 seeds the harness; #193 adds missing phrase implementations). Not one merged module. Naming: Prefer Activity numeric IDs (#188–#195) in cross-references; TFK-0N labels are workflow abbreviations only. --- ## Clarifications (architecture contract) 1. Shared catalog, separate implementations. The Step Library Catalog (phrases/vocabulary) is shared between AT and E2E. Python step modules are not shared: AT in `docs/features/steps/` (Django test client); E2E in `tests/e2e/steps/` (Playwright). 2. "Same Gherkin phrases" means catalog alignment, not one code tree. Missing phrases → Activity #193 (TFK-07). 3. No activity reorder. Placement after #191 is intentional: AT stack first, E2E harness second. #192 does not unblock #189–#191. --- ## Key Concepts ### E2E vs Acceptance Tests \| Aspect \| Acceptance Test (AT) \| E2E Test \| \|--------\|---------------------\|----------\| \| Focus \| Proving ONE feature works \| Fulfilling a GOAL using system means \| \| Path \| `docs/features/` \| `tests/e2e/` \| \| Engine \| Django test client (`behave --simple`) \| Playwright + LiveServer \| \| Scope \| Single .feature, single scenario \| Multi-phase .feature, sequential \| \| Speed \| Fast (no browser) \| Slow (real browser, screenshots) \| \| When \| Every CI run \| Staging before release \| ### Big Idea: Certify Any Environment Using `--base-url=...` we can point Playwright at any place where the application runs: - `--base-url=http://localhost:8000` — local development - `--base-url=https://staging.example.com` — staging - `--base-url=https://app.example.com` — production smoke test Similarly `--db-uri=...` points where initial setup has to be loaded. --- ## Process ### 1. E2E Configuration ```bash make test-e2e BASE_URL=https://staging.example.com DB_URI=postgres://... # → manage.py behave tests/e2e/ ``` ### 2. E2E Feature Structure Each E2E test is written as one long multi-phase `.feature` file under `tests/e2e/` running sequentially. ### 3. Screenshot on Every Step `tests/e2e/environment.py` captures a screenshot after every step for visual verification and debugging. --- ## E2E Session Authentication (mandatory) When E2E steps require a logged-in browser session on a Django app: 1. Seed or factory-create the user in the test database. 2. Use Django test `Client().force_login(user)` — read the `sessionid` cookie. 3. Inject the cookie into Playwright via `page.context.add_cookies([...])` before navigating to protected routes. 4. Provide a shared helper in `tests/e2e/steps/` (e.g. `force_login_playwright`) — do not duplicate per feature. Anti-patterns: manual `SessionStore`; assuming UI login flow exists before AUTH screens ship; AT-only auth patterns copied without cookie injection. Reference: Skill 45 Pattern 4 (Behave-Django BDD Runner). Seed this helper when bootstrapping the E2E harness in this activity.
Construction 3
7	Add Step and/or Fixture to Library	TFK-7	Add Step and/or Fixture to Library Objective On-demand activity invoked when ESM-05 "Write Feature Files" …	Read the 'TFK-7 Add Step and/or Fixture to Library' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Add Step and/or Fixture to Library ## Objective On-demand activity invoked when ESM-05 "Write Feature Files" or BPE cannot assemble a scenario from existing steps/fixtures. Creates new reusable step definitions and/or fixtures following library conventions, then updates the catalogs. --- ## Trigger Called by other workflows when: 1. ESM-05 is writing a .feature and cannot find a matching step in the Step Library 2. BPE-04 needs a fixture preset that doesn't exist 3. A new domain area requires new step patterns 4. BPE-02 / MIN-04 declares a PRF-* row in SAO §17 but `tests/fixtures/llm_scripts/<prf-id>/` has no happy/reject script — add ScriptedLLM queue fixture(s) and catalog row (lane 1; pytest, not Gherkin) 5. DTA-19 / BPE adds a TASK-* golden row but `tests/fixtures/agent_tasks/<task-id>/` is missing input corpus, `expected.yaml`, or oracle `meta.yaml` — add frozen task fixture and catalog row (lane 4) 6. An existing PRF script needs a new reject_* variant (429, parse fail, HITL block, exhausted queue) per artifact 56 adverse sketch ## Process ### 1. Assess What's Needed - Identify the gap: missing step pattern, missing fixture, or both - Check if an existing step can be generalized/parameterized instead of creating a new one - For PRF/TASK triggers: confirm the gap is fixture/oracle catalog — not a missing Gherkin step (see TFK-03 footnote) ### 2. Create Step Definition (if needed) Follow library conventions: - Place in appropriate domain file (navigation, forms, tables, auth, assertions) - Use `data-testid` selectors - Parameterize generically - Log actions at INFO level Skip step creation when trigger is PRF script or TASK golden only. ### 3. Create Fixture (if needed) Follow library conventions: - JSON for static reference data → `tests/fixtures/presets/` - FactoryBoy for dynamic data → `tests/fixtures/factories/` - PRF ScriptedLLM queue → `tests/fixtures/llm_scripts/<prf-id>/` (`happy.json`, `reject_.json`) - TASK golden task* → `tests/fixtures/agent_tasks/<task-id>/` (`input/`, `expected.yaml`, `meta.yaml` per TFK-04 oracle schema) - Document scope and dependencies ### 4. Update Catalogs - Add new step to Step Library Catalog (GUI/AT only) - Add new fixture to Fixture Library Catalog - For agent fixtures: include `prf_id` / `task_id`, `oracle_type`, `pass_band`, `linked_test`, `sao_ref` ### 5. Verify - Write a small test scenario using the new step/fixture - Run it to verify it works - Ensure existing tests are not broken - PRF: run `pytest -m agent_proof` on the linked test; TASK: run `pytest -m quality` on the linked test (local or `agent-eval.yml` — not default PR until lane rules say so)
8	Validate Test Health	TFK-8	Validate Test Health Objective Periodic health check of the test suite: analyze coverage quality (not …	Read the 'TFK-8 Validate Test Health' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Validate Test Health ## Objective Periodic health check of the test suite: analyze coverage quality (not just quantity), flag anti-patterns, identify tests that need strengthening, and verify all test levels pass. --- ## Process ### 1. Run Coverage Analysis ```bash make test-coverage ``` Review coverage report, but remember: coverage is meaningless without quality. ### 2. Quality Audit — Flag Anti-Patterns Check for typical mistakes: Useless Tests: - Testing default Model methods (`model.save()`, `model.delete()`, `model.__str__()`) - Testing Django auto-generated views without custom logic - Tests that only check return codes without verifying side effects Testing Implementation, Not Behavior: - Test checks `save_changes()` returns OK but never verifies changes actually persisted - Test is tightly coupled to implementation details Fat Controller Signal: - Logic in views/controllers instead of services → tests are hard to write - Guideline: Services do the thinking → Views package and provide formatted data → Templates render it Mocking Violations: - Any mock/patch in integration tests → violation of no-mock rule ### 3. Verify All Test Levels Pass ```bash make test # unit + integration make test-at # acceptance make test-e2e # E2E (if staging available) ``` When SAO §17 applies: ```bash make test-agent-proof # lane 1 (when AGENTS_ENABLED=true) make test-agent-live # lane 3 (nightly / manual) make test-agent-quality # lane 4 (promotion band) ``` ### 4. Produce Health Summary Categorize findings as CRITICAL / WARNING / INFO. --- ## Agent eval health audit (when SAO §17 applies) Skip this subsection when SAO §17 is N/A. ### FakeLLM / ScriptedLLM script hygiene \| Check \| Severity \| Action \| \|-------\|----------\|--------\| \| FakeLLM or test double returns `[]` / empty string on queue exhaustion \| CRITICAL \| Replace with explicit raise; add `reject_exhausted` script variant \| \| Script turn count < PRF test's expected agent loop iterations \| WARNING \| Extend `llm_scripts/<prf-id>/happy.json` \| \| Live LLM used in `@agent_proof` test \| CRITICAL \| Move to `@live_llm` or switch to ScriptedLLM \| \| `assert_agent_story` beats missing for declared PRF row \| CRITICAL \| Add happy/reject test per rule `do-assert-agent-story` \| ### Tool / domain mocks in `@agent_proof` \| Check \| Severity \| Action \| \|-------\|----------\|--------\| \| `@patch` on ToolExecutor, broker, or domain service in `@agent_proof` \| CRITICAL \| Remove patch — lane 1 requires real DB + real tool registry \| \| Asserting assistant chat prose instead of trace beats \| WARNING \| Refactor to `assert_agent_story` control-plane keys \| \| Lane 2 deterministic test uses ScriptedLLM \| INFO \| Use plain pytest — no LLM for D0/parse/validate \| Carve-out: only CAP-004 ScriptedLLM may stand in for the LLM in integration tests (rule `do-not-mock-in-integration-tests` ALTER). ### PRF coverage vs SAO §17 Build a matrix from SAO §17 PRF checkbox table vs repo: \| SAO §17 PRF row \| `llm_scripts/` present \| `@agent_proof` test \| happy + reject \| Status \| \|-----------------\|------------------------\|---------------------\|----------------\|--------\| \| e.g. PRF-SC01-04 \| yes/no \| yes/no \| yes/no \| PASS / GAP \| CRITICAL gaps: in-scope PRF row with no linked test or missing reject variant when artifact 56 marks adverse. WARNING gaps: test exists but catalog row in Fixture Library missing `sao_ref` / `linked_test`. ### TASK coverage vs SAO §17 (lane 4) \| SAO §17 TASK row \| `agent_tasks/` oracle \| `@quality` test \| pass_band documented \| Status \| \|------------------\|----------------------\|-----------------\|----------------------\|--------\| \| e.g. TASK-SC02-01 \| yes/no \| yes/no \| yes/no \| PASS / GAP \| Lane 4 gaps are WARNING until promotion window — do not fail PR CI for missing TASK tests. Include agent audit findings in the TAF-08 health summary with lane tags (L1–L4) for TFK-09 scoreboard input.
9	Prepare Test Report	TFK-9	Prepare Test Report Objective Analyze test coverage, last run results, and TAF-08 health findings to …	Read the 'TFK-9 Prepare Test Report' activity from playbook 'Junior Edda' (workflow 'Test Automation Framework'). Follow its guidance when assisting me. # Prepare Test Report ## Objective Analyze test coverage, last run results, and TAF-08 health findings to produce an actionable test report. Outcomes are turned into GitHub Issues assigned for implementation — closing the feedback loop between test health and development work. --- ## Process ### 1. Gather Inputs - TAF-08 "Validate Test Health" findings - Last CI/CD run results (pass/fail per stage, duration, flaky test history) - E2E screenshot archive - `tests.log` analysis When SAO §17 applies, also gather: - Last `agent-eval.yml` run (lanes 3–4): pass/fail, provider, model pin, duration - Lane 1 PR `test-agent-proof` history when `AGENTS_ENABLED=true` - TFK-08 PRF/TASK coverage matrix vs SAO §17 ### 2. Analyze Coverage Gaps For each area with insufficient coverage: - Identify which scenarios from .feature files lack corresponding tests - Identify services/methods with zero or low coverage - Assess risk and prioritize For agent projects: cross-check SAO §17 PRF/TASK rows against green tests (TFK-08 matrix). ### 3. Analyze Quality Issues \| Category \| Examples \| Issue Type \| \|----------\|----------\|------------\| \| Missing tests \| Service method with no test \| Scenario issue \| \| Weak tests \| Tests check return code but not side effects \| Enhancement issue \| \| Fat controllers \| Logic in views needing extraction \| Refactoring issue \| \| Mocking violations \| Mocks in integration tests \| Bug issue \| \| Flaky tests \| Non-deterministic pass/fail \| Bug issue \| \| Agent proof gaps \| PRF row unchecked vs repo \| Scenario issue (lane 1) \| \| FakeLLM hygiene \| Silent queue exhaustion \| Bug issue (lane 1) \| \| Task band regression \| recall below certified pass_band \| Enhancement issue (lane 4) \| ### 4. Produce Report and Create Issues For each finding, create a GitHub Issue with: - What: specific test/file/scenario that needs work - Why: what risk the gap exposes - How: concrete action - Effort: easy/medium/hard - Priority: critical/important/nice-to-have --- ## Agent eval scoreboard (lanes 3 & 4) When SAO §17 applies, append a Lane 3/4 scoreboard section to the test report. Omit when agents N/A. ### Lane 3 — Live contract eval \| Run id \| Date \| Provider / model \| Tests run \| Pass \| Fail \| Flake \| Notes \| \|--------\|------\|------------------\|-----------\|------\|------\|-------\|-------\| \| … \| … \| e.g. anthropic/claude-* \| `@live_llm` count \| \| \| \| temp=0; contract asserts only \| Track: tool-call envelope shape, rate-limit handling, timeout bounds. Not a merge gate — trend for infra/prompt regressions before promotion. ### Lane 4 — Task performance quality \| Task id \| Oracle type \| Pass band \| Last N runs \| Recall / metric \| Extras \| Certified \| vs prior cert \| \|---------\|-------------\|-----------\|-------------\|-----------------\|--------\|-----------\|---------------\| \| TASK-SC02-01 \| set_recall \| ≥0.95, extras≤1 \| 3 \| 0.97 \| 0 \| ✅ \| +0.02 \| Certification rule: promote only when all in-scope TASK rows meet `pass_band` for `n_runs` consecutive evals at temp=0. Regression vs last certified band → open GitHub Issue (priority: important) with task id and metric delta. ### Combined agent summary (for PIN / release notes) \| Lane \| CI home \| Last green \| Open gaps (from TFK-08) \| Release impact \| \|------\|---------\|------------\|---------------------------\|----------------\| \| 1 PRF \| optional PR step \| … \| … \| blocks merge if enabled + red \| \| 2 shell \| `ci.yml` integration \| … \| … \| blocks merge \| \| 3 contract \| `agent-eval.yml` \| … \| … \| advisory / optional CD \| \| 4 quality \| promotion band \| … \| … \| blocks promote if cert required \| Production signals (outside pytest): optionally note HITL reject rate, edit-before-approve ratio, learning-rule growth — feeds PIN-02; reference skill Agent Task Eval § Production quality. Feed scoreboard CRITICAL/WARNING rows into §4 issue creation with labels `agent-lane-3`, `agent-lane-4`, or `agent-prf-gap` as appropriate.

Activities in Test Automation Framework