Playbook Recapture Ledger
Document
Description
Playbook Recapture Ledger
Playbook 45 draft. Recapture from FeatureFactory-io opsdl, 2026-09-14–15, plus KMS Bucket Keys the same day, plus Jupyter Athena login / mart-tab query recapture 2026-09-17/18.
- Status on every row is the live hosted truth, not the first ALTER draft.
- Never re-apply the Historical ALTER trail that used to live in
docs/playbook-change-ledger.md. - Deferred rows stay deferred (do not mark done).
Each row: ID | status | finding | hosted objects (live)
2026-09-17/18 Jupyter Athena + mart tabs
| ID | Status | Finding | Hosted objects |
|---|---|---|---|
| X-16 | fixed | AWS login refresh is not Athena AccessDenied: CreateOAuth2Token / LoginRefreshRequired expired token; wrangler never reached StartQueryExecution; running Jupyter kernel keeps the dead token |
Skills 66, 70, 114; activities 398, 254; aws login then restart kernel; botocore[crt] in requirements-eda.txt |
| X-17 | fixed | Mart tabs must query Athena when contract views exist; print("Instantiate: …") stubs are a fail; skip-when is missing views only |
Skills 114, 115; activity 254; shared streamlit/tab_athena.py; pin QueryExecutionId; empty cohort names the window |
| X-18 | fixed | Empty promoted_eda.yaml rows: [] is correct until Look-until-Trusted survivors; TSY floor still cycle + quadrant + deficiency register |
Activity 254; skill 114; do not dump Metric Render Catalog |
2026-09-15 KMS
| ID | Status | Finding | Hosted objects |
|---|---|---|---|
| X-15 | fixed | SSE-KMS without S3 Bucket Keys: every Put/Get is a KMS request; one-file-per-event Parquet + Glue/Athena burned 20k free KMS requests on 2026-09-15 | Rule 114 s3-bucket-keys-with-sse-kms; skill 74 recipe 4 + pitfall; skill 64 recipe 6 + pitfall; activity 256 step 3 + Done-when; activity 244 step 5 + Done-when; rule 66 (SSE-KMS and Bucket Keys); IngestionStack bucket_key_enabled=True |
2026-09-15 execution recapture
| ID | Status | Finding | Hosted objects |
|---|---|---|---|
| X-01 | fixed | GitHub OIDC sub is ID-qualified repo:Org@ID/repo@ID:ref:... |
Rule 110; skill 75; activity 257; DeployStack trusts name-only and repo:org@*/repo@*:* |
| X-02 | fixed | GitHub rejects repo secrets named GITHUB_* |
Rule 111; repo secret OPSLAKE_GITHUB_PAT; GHA secret AWS_DEPLOY_ROLE_ARN; populate-lake syncs PAT to Secrets Manager |
| X-03 | fixed | App Runner has no Streamlit WebSockets (/_stcore/stream 403) |
EdaStack ECS Fargate + ALB; activity 250 renamed; skill 69 rewritten; rule 113; artifact 151 |
| X-04 | fixed | Dockerfile copied only app.py → ModuleNotFoundError: glue_catalog |
Skill 69: COPY *.py and pages/ |
| X-05 | fixed | wr.catalog.tables hid Glue views |
streamlit/glue_catalog.py paginated GetTables (skill 69 / mart) |
| X-06 | fixed | ECS task role lacked curated s3:ListBucket |
Skills 74 / 69; deploy.yml IAM simulate |
| X-07 | fixed | Issues API default state=open → 0 closed → empty cycle time |
Activity 393 Backfill; skill 63 state=all; skill 111 Populate the Lake; artifact 150 |
| X-08 | fixed | source in Parquet body + Hive partition → Glue duplicate columns |
Skill 64; rule 112; parquet_transform.py _normalize_row omits source |
| X-09 | fixed | Skill 65 said crawler-on-S3-write; CDK uses daily cron | Skill 65; activity 245 cron(0 6 * * ? *); populate-lake one-shot |
| X-10 | fixed | Backfill envelope {payload:{...}}; feature filter was Feature/LOG1.1 only |
Skill 66 $.payload.*; skill 67 classifier + BPE phases |
| X-11 | fixed | Four stacks not two | Skill 74; activity 256; artifact 149 OIDC role; stacks Deploy + Ingestion + Catalog + Eda |
| X-12 | deferred | S-04 agent blob still inlined on activity export | No playbook object change |
| X-13 | deferred | Org webhook registration still a human post-deploy step | Activity 243 / skill 63 honesty; not marked done |
| X-14 | deferred | GithubProjectsPollFn still heartbeat |
Skills 63 / activity 243 honesty notes |
| E-06 | fixed | Mart is ECS Fargate + ALB, not App Runner (supersedes 2026-09-14 App Runner wording) | Same hosted objects as X-03 |
Closed 2026-09-15 (not separate IDs): PAT in Secrets Manager via populate-lake; AWS_DEPLOY_ROLE_ARN set; crawlers/Athena proof via .github/workflows/populate-lake.yml.
2026-09-14 structural rows
| ID | Status | Finding | Hosted objects |
|---|---|---|---|
| P-01 | fixed | IaC workflow order=5 while BPE consumes CDK |
Rule 109 execute-by-step-labels; workflows named 1.…5. |
| P-02 | fixed | No Scope / Preflight activity | Activity 390; skill 106; artifact 142; docs/lake-scope-contract.yaml |
| P-03 | fixed | GitHub Actions predecessor-linked to GitLab CI | XOR on contract ci; activity 258 predecessor is 256 (not 257) |
| P-04 | fixed | Catalog hardcoded as four GitLab/Jira/GitHub views | Views = Scope Contract list; this run five GitHub views; no gitlab/jira |
| P-05 | fixed | playbook.md was a file list |
Every activity Purpose / Inputs / Steps / Done when / Feeds |
| C-01 | fixed | Three different S3 layouts | Activity 244; skill 64; rule 65; raw {source}/{yyyy}/{mm}/{dd}/ vs curated {table}/source=/dt= |
| C-02 | fixed | Artifact 87 said SAM | Artifact 87 CDK-modeled Lambdas, not SAM |
| C-03 | fixed | Skill 74 hard-coded TypeScript paths | Skill 74 Python infra/app.py default |
| C-04 | fixed | Partition rule example source=gitlab/dt= |
Rule 65 {table}/source={source}/dt= |
| S-01 | fixed | Skills named files with no templates | Copy-this on Scope, CDK, GHA, ingestion, Parquet, Glue, Athena |
| S-02 | fixed | No pytest synth story | tests/unit/test_*_stack.py + test_scope_contract.py |
| S-03 | fixed | GHA activity produced no artifact | Artifacts 143 / 144; .github/workflows/deploy.yml; no .gitlab-ci.yml this run |
| S-04 | deferred | Agent definitions pasted into every activity export | Same as X-12 |
| H-01 | fixed | Phase order disagreed with workflow.order |
STEP labels match phases; ignore workflow.order |
| E-01 | fixed | DeployStack OIDC + lake_repo / oidc_sub |
DeployStack; scope contract keys; artifact 149 |
| E-02 | fixed | Backfill, Secrets Manager, pyarrow, daily crawlers | Activities 393 / 243 / 244 / 245; skill 111 |
| E-03 | fixed | STEP 3–5 docs, Jupyter, QS/Huginn deferrals | Activities 247–255; artifacts 91 optional QS, 94 Huginn deferred |
| E-04 | fixed | pytest synth + skip_bundling |
Skill 74 pytest recipe |
| E-05 | fixed | Bootstrap + sandbox mint; repo on GitHub; GHA secret | Activity 257 chicken-and-egg; laptop = bootstrap + one OpsLakeDeploy mint |
| E-07 | fixed | First App Runner image was arm64 from Mac | Skill 69 / EdaStack Platform.LINUX_AMD64 (now on ECS, same pin) |
| (no ID) | fixed | Customer-managed kms.Key on S3+SQS → CFN cycle LakeKey -> RawBucket -> LakeKey |
BucketEncryption.KMS_MANAGED + X-15 Bucket Keys |
Stale ALTER trail (do not copy wording; superseded)
| Sidecar text | Status | Live truth |
|---|---|---|
Activity 244 pending: awswrangler layer |
stale | Skill 64 is pyarrow in Lambda asset |
Activity 245 pending: EventBridge rate 1 day as the only story |
stale | Daily cron + populate-lake one-shot |
Activity 243 executed: scripts/invoke_backfill.py |
stale | .github/workflows/populate-lake.yml |
Activity 245 executed: scripts/run_crawlers_once.py as proof |
stale | populate-lake.yml |
Activity 246 executed: scripts/create_athena_views.py as proof |
stale | populate-lake.yml Athena LIMIT 10 |
| Activity 250 App Runner name/skill/artifact | stale | X-03 / E-06 ECS Fargate + ALB |
| Activity 250–252: Streamlit local mart | stale | ECS mart via GHA |
ADD Artifact OIDC: trust main branch only |
stale | Artifact 149 trusts both sub shapes |
| ALTER Artifact 143 confused with CDK app artifact 95 | stale | Do not retarget |
| Duplicate ADD Backfill (pending vs executed) | stale | Activity 393 + skill 111; see X-07 |
Hosted-object index
- Activity 393 Backfill Historical GitHub Events
- Skill 111 Populate the Lake
- Rules 109–114 (STEP labels, OIDC both subs, GHA
GITHUB_*ban, partition keys out of body, Streamlit ECS+ALB, S3 Bucket Keys) - Artifacts 142 Scope Contract, 143 GHA pipeline, 144 GitLab pipeline (skip this run), 149 OIDC role, 150 Populate-Lake, 151 Streamlit mart, 152 Feature Cycle Time dashboard
Metadata
- Type:
- Document
- Required:
- No
- Created:
- Sep 16, 2026
- Updated:
- Sep 18, 2026
Producer Activity
Model the Data Lake as CDK Stacks
Model and Deploy Infrastructure as Code
Consumer Activities 0
No activities consume this artifact yet.