Activities - Delivery Constraint & Opportunity Analysis

Activities in Delivery Constraint & Opportunity Analysis

All activities across workflows in this playbook

Activities 17

Name	Abbrev.	Workflow	Phase	Actions
Scope Sources, Org, and Environment	MCE-1	Model and Deploy Infrastructure as Code	Datalake Formation	Read the 'MCE-1 Scope Sources, Org, and Environment' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Write the Lake Scope Contract before any AWS resource is modeled. Later activities read only this file for org, sources, CI host, AWS account, region, buckets, Athena view names, lake repo, OIDC sub, and secret keys. ## Inputs - Artifact Operations Lake Starter Kit — clone `https://github.com/FeatureFactory-io/ops-lake-starter` (Use this template). Not the live opsdl lake. - Human target (GitHub org / GitLab group / Jira site). - `aws sts get-caller-identity` and configured region. - Where the lake repo will live (GitHub vs GitLab). - Skill Scope the Operations Lake → Copy-this YAML (and Jira flow keys if jira is scoped). ## Steps 1. Clone the starter kit into the org (GitHub → Use this template). Rename `lake_repo` if the clone is not `ops-lake-starter`. Open skill Scope the Operations Lake. 2. Fill `docs/lake-scope-contract.yaml` in the clone: `org`, `host`, `sources`. List repos: `gh api orgs/{org}/repos --paginate`. Projects v2: `gh api graphql -f query='query($org:String!){organization(login:$org){projectsV2(first:20){nodes{id title}}}}' -f org=YOUR_ORG`. Default github entities = commits, pull_requests, issues, workflow_runs, project_items. Drop unused. Do not add gitlab or jira unless this org uses them. 3. Set `ci` to exactly one of `github_actions` \| `gitlab_ci` matching `host`. 4. Fill `aws_account`, `aws_region`, `env`, `lake_name`, `lake_repo`, `oidc_sub`. Derive `views` as `{source}_{entity}` (copy-this list is in the YAML). 5. Fill `secrets.github_webhook` and `secrets.github_api_token`. GitHub Actions PAT secret must not be named `GITHUB_` — use `OPSLAKE_GITHUB_PAT`. If `jira` is in `sources`, copy skill 106 Jira flow keys* (project keys, started_statuses, jira views) — otherwise ignore skill Jira Flow Extract. 6. Canonical paths in the template (raw JSON vs curated Hive Parquet). Two buckets, always. Keep `streamlit/data/contract_views.yaml` in sync with `views` so mart tabs skip correctly. ## Done when - Starter kit cloned (not opsdl). - `docs/lake-scope-contract.yaml` is filled and committed. - `views` is the complete catalog surface later activities will create. - `ci` is XOR; `lake_repo` and `oidc_sub` are set. ## Feeds Activity Model the Data Lake as CDK Stacks.
Model the Data Lake as CDK Stacks	MCE-2	Model and Deploy Infrastructure as Code	Datalake Formation	Read the 'MCE-2 Model the Data Lake as CDK Stacks' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Model every AWS resource the lake needs as a Python CDK app (TypeScript only if the host repo is already TypeScript). No console clicks. No separate SAM template. ## Inputs - Starter clone `infra/` already has four stacks. Customize from the filled contract; do not scaffold empty. - `docs/lake-scope-contract.yaml`. Instantiate only sources/tables listed there. - Skill AWS CDK Infrastructure as Code → Recipe / Copy-this tests. ## Steps 1. Open skill AWS CDK Infrastructure as Code. Files to touch: `infra/ops_lake/scope.py` (loader), then `deploy_stack.py`, `ingestion_stack.py`, `catalog_stack.py`, `eda_stack.py` only as the contract requires. Do not scaffold empty `infra/`. 2. `DeployStack`: GitHub (or GitLab) OIDC + deploy IAM role. Trust both name-only and ID-qualified `sub` patterns (skill GitHub Actions CI/CD → Copy-this IAM trust). Output `DeployRoleArn` for `AWS_DEPLOY_ROLE_ARN`. 3. `IngestionStack`: raw + curated + athena-results buckets (names from contract), `BucketEncryption.KMS_MANAGED` and `bucket_key_enabled=True`, lifecycle; HTTP API; one webhook Lambda per scoped source; `GithubBackfillFn` (10 min / 1024 MB); EventBridge poller for Projects v2 if github scoped (heartbeat until GraphQL ships); DynamoDB cursor; SQS; `ParquetTransformFn`. Do not attach a customer-managed `kms.Key` to buckets that notify SQS. If jira scoped, wire `jira_flow.py` copied from `infra/lambdas/jira_flow.py.example`. 4. `CatalogStack`: Glue database `athena_database`; one crawler per contract `views` on `s3://{curated}/{table}/`; Athena workgroup; named preview queries matching `views`. 5. `EdaStack`: Streamlit on ECS Fargate + ALB (`linux/amd64`). Dockerfile COPY `.py`, `pages/`, and `data/`. Grant task role Athena, Glue GetTables, curated-bucket read, Athena-results read/write. Output `StreamlitMartUrl`. 6. Environment from CDK context / SSM. Synth: from repo root `cdk synth -c skip_bundling=true` (or `cd infra` if that is how this clone is wired). 7. Pytest: `tests/unit/test_deploy_stack.py`, `test_ingestion_stack.py`, `test_catalog_stack.py`, `test_eda_stack.py`, `test_story_data.py`. Assert BucketKeyEnabled on all three lake buckets. ## Done when - `infra/app.py` instantiates all four stacks. - Synth tests pass, including S3 Bucket Keys. - GitHub-only contract → no GitlabWebhookFn / live JiraPollFn. ## Feeds XOR: Deploy via GitHub Actions* or Deploy via GitLab CI.
Deploy via GitHub Actions	MCE-3	Model and Deploy Infrastructure as Code	Datalake Formation	Read the 'MCE-3 Deploy via GitHub Actions' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Deploy the CDK app through GitHub Actions with OIDC. Production path for a GitHub-hosted lake repo. ## Inputs - Lake Scope Contract (`ci` must be `github_actions`). - CDK Application (includes DeployStack). - Starter `.github/workflows/deploy.yml` — copy from the clone; do not invent a second pipeline. Template: `https://github.com/FeatureFactory-io/ops-lake-starter/blob/main/.github/workflows/deploy.yml`. - Skill GitHub Actions CI/CD → Recipe. ## Skip when `ci` is `gitlab_ci`. Do not create deploy.yml "just in case". ## Steps 1. Keep starter `.github/workflows/deploy.yml`. 2. Chicken-and-egg: laptop, same region as the contract: `cdk bootstrap` then `cdk deploy OpsLakeDeploy` (or `--all`) once. Read `DeployRoleArn` from stack outputs. GitHub → Settings → Secrets and variables → Actions → New repository secret `AWS_DEPLOY_ROLE_ARN`. Also `OPSLAKE_GITHUB_PAT` (`repo`, `read:org`). Never `GITHUB_`. After that, never `cdk deploy` a shared env from a laptop. 3. `permissions: id-token: write`. Assume the deploy role via `aws-actions/configure-aws-credentials`. No `AWS_ACCESS_KEY_ID`. 4. Jobs: pytest → `cdk deploy --all --require-approval never`. Print `DeployRoleArn`, `GithubWebhookUrl`, `StreamlitMartUrl` to the job summary. IAM simulate Streamlit task-role S3 read. 5. OIDC trust both* `repo:{org}/{lake_repo}:` and `repo:{org}@/{lake_repo}@:`. A `main` push AssumeRoleWithWebIdentity must succeed (check the Actions log). 6. After first GHA deploy, record URLs/ARNs in `docs/deploy-manifest.yaml`. Org webhook registration is the next human step (activity Stand Up Serverless Ingestion) — deploy does not by itself land events. ## Done when - Workflow file is the only shared-env deploy path. - OIDC covers both `sub` shapes; `AWS_DEPLOY_ROLE_ARN` is set. ## Feeds Pipeline + populate-lake. Shared-env resources appear only after this workflow runs on `main`.
Deploy via GitLab CI	MCE-3	Model and Deploy Infrastructure as Code	Datalake Formation	Read the 'MCE-3 Deploy via GitLab CI' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Deploy the CDK app through GitLab CI with OIDC. Alternative to GitHub Actions. ## Inputs - Lake Scope Contract (`ci` must be `gitlab_ci`). - CDK Application. - Starter `.gitlab-ci.yml.example` — copy to `.gitlab-ci.yml`. Skill GitLab CI/CD → Recipe. ## Skip when `ci` is `github_actions`. Running both against one AWS account causes split-brain deploys. ## Steps 1. Copy `.gitlab-ci.yml.example` to `.gitlab-ci.yml`. Do not also keep a live GitHub `deploy.yml` against this account. 2. Chicken-and-egg: laptop `cdk bootstrap` then one `cdk deploy OpsLakeDeploy` to mint the GitLab OIDC IAM role. Store the role ARN as CI/CD variable `AWS_DEPLOY_ROLE_ARN` (not `AWS_ACCESS_KEY_ID`). Trust `aud` = `https://gitlab.com` (or this instance URL) and the project `sub`. 3. `id_tokens` block mints the OIDC token. pytest → `cdk synth` → `cdk diff` → `cdk deploy` with `when: manual` for production. Register GitLab Environment `production`. 4. After first deploy, record URLs/ARNs in `docs/deploy-manifest.yaml`. ## Done when - `.gitlab-ci.yml` is the only shared-env deploy path. - GitHub Actions deploy workflow is absent. - A `main` pipeline assumed the role. ## Feeds Pipeline (populate equivalent on GitLab if you add it; otherwise GitHub populate-lake is the GitHub-ci path).
Stand Up Serverless Ingestion from Scoped Sources	BPE-1	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-1 Stand Up Serverless Ingestion from Scoped Sources' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose Land raw JSON events from scoped sources into the raw bucket. Implement only Lambdas the Scope Contract lists. Org webhook registration is a human GitHub (or GitLab) Settings step after the first CI deploy — deploy does not by itself land events. ## Inputs - Lake Scope Contract (`sources`, `views`, bucket names, `secrets.github_webhook`). - CDK Application (IngestionStack already declares the functions). Gate is the artifact — predecessors cannot cross workflows. - Stack output `GithubWebhookUrl` (job summary after `deploy.yml`, also `docs/deploy-manifest.yaml`). - Skill Serverless Data Ingestion → Recipe. ## Steps 1. Open skill Serverless Data Ingestion. Handlers already live under `infra/lambdas/` in the starter (`github_webhook.py`, and `jira_flow.py` only after you copy `infra/lambdas/jira_flow.py.example`). Edit signatures / secret names from the contract; do not scaffold empty. 2. Confirm Secrets Manager `ops-lake/github-webhook-secret` exists (CDK creates it; populate-lake may sync). Value must match the GitHub webhook secret you will paste in org Settings. 3. Register the org webhook (human): GitHub org → Settings → Webhooks → Add webhook. Payload URL = `GithubWebhookUrl`. Content type `application/json`. Secret = the Secrets Manager value. SSL verify. Events: Issues, Pull requests, Pushes, Workflow runs. Save. Send a test ping. 4. GitHub handler: `GithubWebhookFn` behind `/webhooks/github`, verify `X-Hub-Signature-256`. Return 200; enqueue SQS for slow work. Prove: CloudWatch `/aws/lambda/GithubWebhookFn`; raw object at `s3://{raw}/github/{yyyy}/{mm}/{dd}/`. 5. GitLab: implement only if `gitlab` is in the contract. Secret `ops-lake/gitlab-webhook-secret`; header `X-Gitlab-Token` (skill 63). 6. Jira: implement only if `jira` is in the contract. Copy `infra/lambdas/jira_flow.py.example` → `jira_flow.py`. The poller lands raw JSON. Historical issues, changelog, and lead/cycle clocks are Backfill Historical Jira Issues and Status History plus skill Jira Flow Extract — a current-status poller is not a cycle-time extract. 7. Write unmodified JSON to `s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json`. No Parquet here. 8. Idempotent on event id. `GithubProjectsPollFn` remains a heartbeat until GraphQL pagination ships — do not join `github_project_items` as a populated Master Roadmap. ## Done when - Org webhook is registered (or GitLab equivalent if that host is scoped). - One working handler per scoped source; zero handlers for unscoped systems. - Test ping produced a raw object (or CloudWatch shows a verified 200). - Secrets live in Secrets Manager / SSM, referenced from CDK. ## Feeds Backfill Historical GitHub Events (when github is scoped), Backfill Historical Jira Issues and Status History (when jira is scoped), then Store Compressed and Partitioned in S3.
Backfill Historical GitHub Events	BPE-2	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-2 Backfill Historical GitHub Events' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose Pull historical GitHub issues, PRs, commits, and workflow runs into the raw bucket so Constraint and Flow Analysis can compute cycle time. Webhooks only catch events after they are registered. ## Inputs - Lake Scope Contract (`repos`, `secrets.github_api_token`). - CDK Application (`GithubBackfillFn` already declared). - Starter workflow `.github/workflows/populate-lake.yml` — template `https://github.com/FeatureFactory-io/ops-lake-starter/blob/main/.github/workflows/populate-lake.yml`. Do not laptop-invoke against shared env. - Skills Serverless Data Ingestion and Populate the Lake. ## Steps 1. Keep starter `infra/lambdas/github_backfill.py` and `populate-lake.yml`. Open skill Populate the Lake. 2. Issues and pulls: `state=all` and `per_page=100`. GitHub's default is open only — that was the empty cycle-time dashboard. Wrap each item `{backfill: true, kind, repo, payload}` with `kind` in `issues` \| `pulls` \| `commits` \| `workflow_runs`. 3. Paginate `Link: rel=next` for list endpoints. Commits API pagination is not the same as issue/PR Link headers — follow `page=` until empty. Timeout 10 minutes / 1024 MB. 4. Repo secret `OPSLAKE_GITHUB_PAT` (`repo`, `read:org`) — never `GITHUB_`. Populate-lake syncs it into Secrets Manager. 5. GitHub → Actions → Populate the Lake* → Run workflow. Watch CloudWatch for `written=N`. Never laptop invoke against shared env. 6. After raw objects land, the same workflow runs `scripts/sync_raw_to_curated.py` then crawlers then Athena proof. Athena LIMIT 10 is activity Present the Catalog via Athena; here you only need `written=N` and raw keys under `s3://{raw}/github/`. ## Done when - Glue `github_issues` will contain closed rows after Parquet + crawl (`payload.state='closed'`), not only open. - CloudWatch shows `written=N` for issues, pulls, commits, workflow_runs. ## Feeds Store Compressed and Partitioned in S3, then Glue/Athena. Cycle-time EDA depends on this history. ## Honesty `GithubProjectsPollFn` is a heartbeat until GraphQL pagination ships. Do not join `github_project_items` as if Master Roadmap items are populated.
Store Compressed and Partitioned in S3	BPE-3	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-3 Store Compressed and Partitioned in S3' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose Turn raw landing-zone JSON into Hive-partitioned Parquet in the curated bucket. ## Inputs - Raw Operational Events (S3 Landing Zone). - CDK Application (`ParquetTransformFn` lives in IngestionStack). - Lake Scope Contract paths: raw vs curated. - Populate-Lake Pipeline (batch transform after backfill). - Skill S3 Data Partitioning and Compression → Recipe step 2 (event→table map). ## Canonical layout (do not invent another) - Raw: `s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json` - Curated: `s3://{curated}/{table}/source={source}/dt={yyyy-mm-dd}/part-{uuid}.parquet` - `{table}` is a name from contract `views`. Two buckets, always. ## Steps 1. Open skill S3 Data Partitioning and Compression. Handler path: `infra/lambdas/parquet_transform.py` (`_normalize_row` omits `source`). 2. Event → table map (skill 64 recipe 2): `push` → `github_commits`, `pull_request` → `github_pull_requests`, `issues` → `github_issues`, `workflow_run` → `github_workflow_runs`; backfill `kind` `commits`/`pulls`/`issues`/`workflow_runs`. Projects v2 → `github_project_items` only when that poller is real. 3. `ParquetTransformFn` writes pyarrow Parquet. Omit `source`/`dt` from the row body — they are Hive partitions only. 4. Webhooks: SQS buffer. Backfill: `scripts/sync_raw_to_curated.py` from populate-lake only, not a laptop against shared env. 5. Lifecycle: hot recent partitions Standard; cold raw/history IA or Glacier. 6. Confirm lake buckets have S3 Bucket Keys (`bucket_key_enabled=True`) via pytest synth (`tests/unit/test_ingestion_stack.py`), not the S3 console encryption tab alone. 7. If `jira` is scoped, run Backfill Historical Jira Issues and Status History (sibling of GitHub backfill) before Glue — this activity's predecessor is GitHub backfill only because workflows cannot cross-link both. ## Done when - Curated prefixes exist for every contract `views` entry that backfill produced. - Glue can crawl without "duplicate columns". - No EMR/Spark cluster. - SSE-KMS Bucket Keys are on (new objects do not call KMS per Put/Get). ## Feeds Catalog with AWS Glue.
Backfill Historical Jira Issues and Status History	BPE-3	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-3 Backfill Historical Jira Issues and Status History' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose When `jira` is in the Scope Contract, pull historical issues plus a status timeline into the raw bucket so Constraint and Flow Analysis can compute lead time and cycle time. A poller that only sees current status cannot reconstruct cycle start. ## Skip when Scope Contract `sources` does not include `jira`. Do not stand up Jira Lambdas or views "just in case". Do not copy `jira_flow.py.example` onto a GitHub-only lake. ## Inputs - Lake Scope Contract (`sources`, project keys from skill Scope the Operations Lake → Jira flow keys, `views`, Started status set, watermark / secret names). - Starter stub `infra/lambdas/jira_flow.py.example` — copy to `infra/lambdas/jira_flow.py` only if jira is scoped. - CDK Application (`JiraPollFn` / Jira backfill function already declared when jira is scoped). - Skill Jira Flow Extract → Extraction recipe. - Skill Serverless Data Ingestion (handler wiring only). ## Steps 1. If jira is scoped: copy `infra/lambdas/jira_flow.py.example` → `jira_flow.py`. Open skill Jira Flow Extract → Extraction recipe. Implement that contract — do not invent a different clock. Secret `ops-lake/jira-api-token`; base URL from contract Jira flow keys. 2. Persist both issue snapshots (`closed_at` = `resolutiondate`) and status segments (`status`, `entered_at`, `exited_at`). 3. JQL is scoped and bounded (`updatedDate`). Cloud: cursor pagination. Server/DC: `startAt`. Page size about 100. 4. Build the timeline from changelog `field == status`. Paginate per-issue changelog when search expand truncates. 5. Record the Started status set on the contract. Compute lead and cycle from the landed columns (business days). Do not re-query Jira in EDA for those clocks. 6. Incremental cursor in DynamoDB keyed by `source`. Periodic bounded backfill for tickets older than the watermark. Populate-lake today is GitHub-shaped — if you add a Jira invoke job, put it in the same workflow; do not laptop-invoke shared env. 7. Write unmodified JSON to `s3://{raw}/jira/{yyyy}/{mm}/{dd}/{event-id}.json`. Parquet is the next activity. ## Done when - Glue (or the contract views) contain resolved rows (`closed_at` not null), not only open issues. - `jira_status_segments` (or the named contract view) has at least one closed segment per resolved issue that has changelog. - Athena proof LIMIT 10 succeeds for every jira view that has Parquet. - Lead/cycle can be computed from curated columns without another Jira call. ## Feeds Store Compressed and Partitioned in S3, then Glue/Athena. Constraint analysis may use these clocks as measured touch/wait inputs; it does not replace impact scoring. ## Honesty Sprint/velocity board reports are out of scope here. Do not treat them as lead or cycle.
Catalog with AWS Glue	BPE-4	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-4 Catalog with AWS Glue' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose Register curated prefixes in Glue so Athena can see tables without hand-written DDL. ## Inputs - Partitioned Parquet Data Lake. - Lake Scope Contract (`athena_database` default `ops_catalog`, `views`). - Populate-Lake Pipeline (one-shot crawler start). Starter: `.github/workflows/populate-lake.yml`. - Skills AWS Glue Cataloging and Populate the Lake. ## Steps 1. Open skills AWS Glue Cataloging and Populate the Lake. Do not start crawlers in the Glue console. 2. One database named from contract `athena_database` (default `ops_catalog`). One crawler per `views` entry, scoped to `s3://{curated}/{table}/`. CDK already declared them — intern does not click Create crawler. 3. Daily schedule `cron(0 6 * * ? )` plus populate-lake job step `start-crawler` after Parquet sync. Wait until State=READY: `aws glue get-crawler --name {table}_crawler` (name from CatalogStack). 4. Built-in Parquet classifier only unless a source fails auto-detect. 5. If Glue reports duplicate columns, go back to Store Compressed and Partitioned in S3* (`source` leaked into Parquet body). Do not patch the crawler. 6. Escalate to Operations Analyst before applying a crawler change that alters an existing column type. ## Done when - Glue tables exist for every contract view that has Parquet (table name = view name). - No crawlers for unscoped systems. ## Feeds Present the Catalog via Athena.
Present the Catalog via Athena	BPE-5	Build the Operational Data Lake Pipeline	Datalake Formation	Read the 'BPE-5 Present the Catalog via Athena' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Build the Operational Data Lake Pipeline'). Follow its guidance when assisting me. ## Purpose Prove the Athena surface. Glue crawler tables are the contract views. Downstream consumers (Streamlit, Pandas, MCP) query those tables (or thin views if you add them), never random Glue names. ## Inputs - AWS Glue Data Catalog. - Lake Scope Contract (`views`, `athena_workgroup` default `ops-lake-analysts`, `athena_database` default `ops_catalog`). - Populate-Lake Pipeline. - Skills Athena SQL Querying and Populate the Lake. - SQL template `sql/proof_views.sql` (placeholders `{database}`, `{view}`). ## Steps 1. Open skills Athena SQL Querying and Populate the Lake. 2. One workgroup from contract `athena_workgroup` (AWS name default `ops-lake-analysts`) with `EnforceWorkGroupConfiguration` and a per-query scan cap (start 5GB). 3. After crawlers READY, populate-lake runs `scripts/create_athena_views.py` — LIMIT 10 per contract view; skip missing tables. Do not laptop-run against shared env. Console alternative: Athena query editor → workgroup dropdown `ops-lake-analysts` → database `ops_catalog`. 4. Copy-this proof: `SELECT COUNT() AS n FROM {view} WHERE source = '{source}'` then `SELECT FROM {view} LIMIT 10`. Backfill JSON: extract `$.payload.`. Prefer closed snapshots when deduping issue numbers. Record each `QueryExecutionId` on artifact Athena Catalog Views. 5. Glue tables already are* the views; the script is a proof runner, not a second catalog of DDL. 6. Hands off to Operations Analyst when all contract views that have data are queryable. ## Done when - Each populated contract view runs a `SELECT ... LIMIT 10` in the workgroup. - Streamlit/Pandas/MCP are pointed at these tables/views only. ## Feeds Constraint and Flow Analysis (artifact gate).
Map the BizDevTestSecOps Value Stream	CAS-1	Constraint and Flow Analysis	Exploratory Data Analysis	Read the 'CAS-1 Map the BizDevTestSecOps Value Stream' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Constraint and Flow Analysis'). Follow its guidance when assisting me. ## Purpose Draw the BizDevTestSecOps map from catalog events that exist, not from a generic GitLab/Jira cartoon. ## Inputs - Queryable catalog views (pipeline done, including closed-issue backfill). - Lake Scope Contract: `docs/lake-scope-contract.yaml` key `views`. - Starter file `streamlit/data/value_stream.yaml` (human notes: `docs/analysis/value-stream-map.md`). - Skill Value Stream and Constraint Analysis → Recipe steps 2 and 4. - Skill Athena SQL Querying → Recipe (workgroup, LIMIT 10, `$.payload.`). ## Steps 1. Open `docs/lake-scope-contract.yaml` → `views`. Those names are* the Glue/Athena tables. 2. Prove each view is queryable: run `scripts/create_athena_views.py` (populate-lake, not a laptop against shared env) or `sql/proof_views.sql` in workgroup `athena_workgroup` (default `ops-lake-analysts`). `SELECT COUNT()` then `LIMIT 10`. Paste `QueryExecutionId` onto the matching row in `streamlit/data/value_stream.yaml`. If n=0 or the table is missing, stop — do not invent a stage. 3. Open `streamlit/data/value_stream.yaml`. Default five stages (skill Value Stream → Recipe step 2): - Biz* — `github_issues` opened (`$.payload.created_at`); join issue number - Dev — `github_pull_requests` opened → merged; `github_commits` in window; join `Closes #N` - Test — `github_workflow_runs` start → first `conclusion=success` - Sec — security-gate job in that run, or a dedicated scan workflow - Ops — deploy workflow run start → complete Delete any row whose `view` is not in `views`. Uncomment gitlab/jira only if those sources exist. Do not use heartbeat `github_project_items` as Biz. 4. Feature-type cohort: skill Value Stream → Recipe step 4 (classifier). Do not map every issue. 5. Commit the filled YAML. That is artifact BizDevTestSecOps Value Stream Map. ## Done when - Map names real view+column pairs for each stage boundary. - Every remaining row has a QueryExecutionId (or an explicit empty-table stop). - No stage points at a table that is not in `views`. ## Feeds Score Cycle Impact.
Score Cycle Impact	CAS-2	Constraint and Flow Analysis	Exploratory Data Analysis	Read the 'CAS-2 Score Cycle Impact' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Constraint and Flow Analysis'). Follow its guidance when assisting me. ## Purpose Reconstruct touch vs wait, score impact, and mark where AI could help. Do not design agents. ## Formula - wall-clock = touch_time + wait_time - effort = people_in_role × touch_time - impact = effort + (people_downstream × wait_time) ## Inputs - Filled `streamlit/data/value_stream.yaml`. - `sql/stage_touch_wait.sql` and skill Cycle Impact Scoring → Recipe. - Skill AI Leverage Mapping → Recipe. - Role capacity file `streamlit/data/role_capacity.yaml`. ## Steps 1. Open `streamlit/data/cost_of_delay.yaml`. For each stage on the map, set `touch_time` and `wait_time` from catalog events (issue → PR → CI → merge → deploy). Never treat wall-clock as effort. Label measured vs assumed. Pin `query_execution_id` from `sql/stage_touch_wait.sql` (workgroup from the contract; extract `$.payload.`). 2. Fill `streamlit/data/role_capacity.yaml`: people_in_role, people_downstream, source of the assumption. Do not* infer headcount from unique GitHub authors. 3. Join issue↔PR on `Closes #N` in payload, never `repo` or timestamps alone. 4. Compute effort, wall-clock, impact (worked sample in the YAML: Sec wait 48h × 8 downstream + 2h touch × 1 person = 386). Keep ToC #1 visible (`toc_rank_1: true`). 5. Fill `streamlit/data/ai_opportunities.yaml`: opportunity names, `ai_can_help` yes/no/unknown, effort S–XL. Low hanging fruit = high impact AND AI-yes AND low effort. No agent products. 6. Those three YAML files are artifacts Delivery Cycle Reconstruction, Cost of Delay Ranking, and AI Opportunity Prioritization Matrix. ## Done when - Ranking has impact scores with measured/assumed labels and one named #1 constraint. - Matrix has at least the high-impact boxes scored; no agent names. ## Feeds Look until trusted, then Triage: Eliminate / Automate / Delegate. ## Skip Do not optimize a non-constraint. Do not put AI on a low-impact stage because it is easy.
Look until Trusted	CAS-3	Constraint and Flow Analysis	Exploratory Data Analysis	Read the 'CAS-3 Look until Trusted' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Constraint and Flow Analysis'). Follow its guidance when assisting me. ## Purpose See the delivery cycle as people live it until the impact ranking can be trusted. This is not a query-console step. ## Inputs - `streamlit/data/cost_of_delay.yaml`, `ai_opportunities.yaml`, `role_capacity.yaml`. - Mart URL: `docs/deploy-manifest.yaml` → `streamlit_mart_url`. - Notebook `notebooks/look-until-trusted.ipynb`. - Skills EDA with Streamlit Data Mart, EDA with Pandas and Jupyter, MCP-Driven Conversational Analytics, Metric Render Catalog → Render pairings / Starter mart tabs. ## Steps 1. Open the mart URL. Home is Delivery cycle, then Opportunity quadrant. Feature Cycle Time (`pages/11_Feature_Cycle_Time.py`) is a check, not the ranking. 2. Look at the whole flow first. Use Metric Render Catalog pairings to challenge the #1 constraint (lead/cycle, throughput, MR hours, CI failure, load). Skip any family whose contract views are missing. 3. Challenge assumed numbers in `role_capacity.yaml`; relabel or remeasure. Jira `status_segments`: touch/wait may be measured (business days). Do not mix with PR hours. 4. Chat/MCP findings are hypotheses until reproduced in `notebooks/look-until-trusted.ipynb` (Restart & Run All). 5. Copy survivors into `streamlit/data/promoted_eda.yaml` (metric id, pairing, unit, n, window, agrees_with_ranking, query_execution_id, tsy_section). Empty file (`rows: []`) is valid if nothing earned TSY — that is not a Present Findings bug. ## Pitfalls - If `LoginRefreshRequired` / `CreateOAuth2Token` expired → `aws login`, restart kernel, not a Glue/Athena ACL bug. Skill Athena SQL Querying / EDA with Pandas and Jupyter. ## Done when - Impact ranking survives a second look (or was corrected in YAML). - Any finding that will inform triage has query text + QueryExecutionId. - Catalog checks that disagree with the ranking are explained or the ranking is updated. ## Feeds Triage: Eliminate / Automate / Delegate.
Triage: Eliminate / Automate / Delegate	CAS-4	Constraint and Flow Analysis	Exploratory Data Analysis	Read the 'CAS-4 Triage: Eliminate / Automate / Delegate' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Constraint and Flow Analysis'). Follow its guidance when assisting me. ## Purpose Decide eliminate vs automate vs delegate for ranked impact constraints, in that order, in writing. Tag cause layers (dE / dK / dI) and route each tag. ## Inputs - `streamlit/data/cost_of_delay.yaml` and `ai_opportunities.yaml`. - Skill Eliminate/Automate/Delegate Triage → Cause tags / Routes. - Template `streamlit/data/action_register.yaml`. ## Steps 1. Open `streamlit/data/action_register.yaml`. One row per ranked constraint, impact order. Fill eliminate first; automate only if eliminate is written no; delegate only if automate cannot encode the judgment. 2. The AI matrix may inform where automate or delegate could apply. It does not skip eliminate and it is not an agent backlog. 3. Automate shape: one Lambda + one Guardrail (policy / allow-list / human approval — name the check, not a product). 4. Tag cause layers on each row (multi-tag allowed). Skill → Cause tags: - dE environment (tool, process, incentive, data, time, authority) - dK knowledge/skill (trainable / selectable) - dI individual (selection, capacity, motivation) "Knows the right move, environment punishes it" stays dE, not dK. 5. Route per tag (skill → Routes): dE+eliminate → drop; dE+process/tooling → process/tooling (not AI SDLC); dE+automate/delegate → AI SDLC candidate; dK → Team Upskill Engine input (stage, clock, unit, n, role); dI → staffing. Do not author TUE modules. 6. Escalate #1 if neither eliminate nor automate works — flow redesign. Do not implement the Lambda in this activity. ## Done when - `action_register.yaml` has decision, cause tags, route, and evidence (clock, unit, n, role) for #1 and every ranked row you touched. - No dK row claims a finished curriculum. No dE process/tooling row is labeled AI SDLC unless automate or delegate was chosen. ## Feeds Tell the Data Story (publish this register — do not re-classify).
Design the Big-Picture-to-Detail Narrative	TSY-1	Tell the Data Story	Presenting Findings	Read the 'TSY-1 Design the Big-Picture-to-Detail Narrative' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Tell the Data Story'). Follow its guidance when assisting me. ## Purpose Plan the stakeholder story: reconstructed cycle first (three metrics), then quadrant, then the deficiency register, then stage detail, then starter tabs. Never lead with a detail chart. Do not re-classify Action Register tags. ## Inputs - YAML in `streamlit/data/` (cost_of_delay, ai_opportunities, action_register, promoted_eda). - Template `docs/analysis/narrative-outline.md`. - Skill Data Storytelling and Dashboard Design → Recipe. - Skill Metric Render Catalog (optional pairings; Streamlit tab map). ## Steps 1. Open `docs/analysis/narrative-outline.md`. 2. Opening visual: Delivery cycle (effort, wall-clock, impact). Then opportunity quadrant. Then deficiency register (finding + decision + dE/dK/dI + receiver). Copy CAS; do not invent tags. - dE + automate/delegate → AI SDLC (receiver only). - Other dE → process/tooling. - dK → Team Upskill Engine input (stage, clock, unit, n, role). - dI → staffing. 3. One section per stage in flow order: finding sentence + Action Register decision (or no action). Embed promoted-eda ids or "none". 4. Cap one visual per finding. Name the clock and n. Do not equate issue business days with MR hours. 5. People: name the role (author / integrator / reviewer). Concentration is a risk. 6. If Present Findings uses Streamlit or Jupyter, list starter tabs to instantiate or skip: Commits, Work items, Changes, Pipelines, Load, Defects, Sprints (skill Present Findings with Streamlit → Starter mart tabs). Do not put a tab first. 7. Name the presentation skill (Jupyter = `notebooks/published-findings.ipynb`; Streamlit = mart pages 0–10). ## Done when - Outline lists cycle, quadrant, register, stage sections, and tabs-to-skip. - Duration KPIs state unit, cohort, and n. - Register beat names receivers only. ## Feeds Present Findings.
Present Findings	TSY-2	Tell the Data Story	Presenting Findings	Read the 'TSY-2 Present Findings' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Tell the Data Story'). Follow its guidance when assisting me. ## Purpose Build the stakeholder story from the outline. Pick one presentation skill. Do not re-derive numbers. Do not re-classify tags. ## Inputs - `docs/analysis/narrative-outline.md`. - `streamlit/data/.yaml`. - Skills: Present Findings with Jupyter* (`notebooks/published-findings.ipynb`), Present Findings with Streamlit (pages 0–10), QuickSight, Power BI. - Skill Metric Render Catalog. ## Steps 1. Choose one skill. Skip QuickSight if unsubscribed. Skip Power BI if no tenant. Do not add BI in CDK. 2. Order: Delivery cycle → Opportunity quadrant → Deficiency register → Stage detail (promoted_eda only) → starter tabs. 3. Streamlit: pages already in the starter (`0_Delivery_Cycle.py` … `10_Sprints.py`). Browser-verify the path end to end, then each instantiated tab. Skip tabs whose contract views are missing (`tab_pages.py`). If the view exists, run Athena via shared `streamlit/tab_athena.py` — do not leave `print("Instantiate: …")` as the tab body. Pin QueryExecutionId; empty cohort names the window. Live GitHub-only lake: instantiate Commits, Work items (volume only — omit lead/cycle without `jira_status_segments`), Changes (MR hours, not issue days), Pipelines (top failing = workflow name; job-level extract is not in the lake), Load (risk signal). Skip Defects and Sprints unless those views exist. 4. Jupyter: open `notebooks/published-findings.ipynb`, kernel Python (opsdl .venv) (`python -m ipykernel install --user --name opsdl --display-name "Python (opsdl .venv)"`; notebook root = repo). Restart & Run All. Skill 114 recipe matches the cell order. 5. Receivers named only: AI SDLC, process/tooling, Team Upskill Engine (candidates), staffing. No TUE modules. No agent names. 6. Write Published Findings Notebook/Dashboard = the mart URL and/or the notebook path. ## Pitfalls - If `LoginRefreshRequired` / `CreateOAuth2Token` expired → `aws login`, restart kernel, not a Glue/Athena ACL bug. - `streamlit/data/promoted_eda.yaml` `rows: []` is still correct until activity 398 promotes Look-until-Trusted survivors. TSY floor is still cycle + quadrant + deficiency register. Do not dump Metric Render Catalog. Do not treat empty promoted-eda as a bug in this activity. ## Done when - A stakeholder can follow cycle → quadrant → register without a debugging session. - Numbers match the YAML ranking files. - Tags match CAS. Omitted tabs listed as skipped for missing views. ## Feeds Configure Huginn for Automated Reporting. ## Skip Do not open with a single-stage chart or Commits tab. Do not name agents to implement.
Configure Huginn for Automated Reporting	TSY-3	Tell the Data Story	Presenting Findings	Read the 'TSY-3 Configure Huginn for Automated Reporting' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Tell the Data Story'). Follow its guidance when assisting me. ## Purpose Keep the published story's underlying data current. Huginn has no MCP connector today. ## Inputs - Published Findings (mart URL or notebook). - Skill Huginn Browser Automation → Recipe. - Starter `docs/huginn/README.md`. ## Steps 1. Read `docs/huginn/README.md`. Check sibling `../huginn` for an existing Agent/Scenario. 2. Huginn UI URL is org-specific — record it in playbook notes. Search Scenarios for lake / Athena / Streamlit. 3. Agent type follows the Present Findings surface (HTTP GET of `streamlit_mart_url` from `docs/deploy-manifest.yaml` if Streamlit). Jupyter has no live URL — defer if that was the surface. 4. Dry-run once. Record Agent name, schedule, purpose. 5. Flag as a future MCP-connector candidate. ## Done when - A scheduled refresh exists, or an explicit deferral is written in `docs/huginn/README.md`. ## Skip when `streamlit_mart_url` is empty — defer and say so.