Activities - Model and Deploy Infrastructure as Code

#	Name	Abbrev.	Guidance	Actions
Datalake Formation 4
1	Scope Sources, Org, and Environment	MCE-1	Purpose Write the Lake Scope Contract before any AWS resource is modeled. Later activities read …	Read the 'MCE-1 Scope Sources, Org, and Environment' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Write the Lake Scope Contract before any AWS resource is modeled. Later activities read only this file for org, sources, CI host, AWS account, region, buckets, Athena view names, lake repo, OIDC sub, and secret keys. ## Inputs - Artifact Operations Lake Starter Kit — clone `https://github.com/FeatureFactory-io/ops-lake-starter` (Use this template). Not the live opsdl lake. - Human target (GitHub org / GitLab group / Jira site). - `aws sts get-caller-identity` and configured region. - Where the lake repo will live (GitHub vs GitLab). - Skill Scope the Operations Lake → Copy-this YAML (and Jira flow keys if jira is scoped). ## Steps 1. Clone the starter kit into the org (GitHub → Use this template). Rename `lake_repo` if the clone is not `ops-lake-starter`. Open skill Scope the Operations Lake. 2. Fill `docs/lake-scope-contract.yaml` in the clone: `org`, `host`, `sources`. List repos: `gh api orgs/{org}/repos --paginate`. Projects v2: `gh api graphql -f query='query($org:String!){organization(login:$org){projectsV2(first:20){nodes{id title}}}}' -f org=YOUR_ORG`. Default github entities = commits, pull_requests, issues, workflow_runs, project_items. Drop unused. Do not add gitlab or jira unless this org uses them. 3. Set `ci` to exactly one of `github_actions` \| `gitlab_ci` matching `host`. 4. Fill `aws_account`, `aws_region`, `env`, `lake_name`, `lake_repo`, `oidc_sub`. Derive `views` as `{source}_{entity}` (copy-this list is in the YAML). 5. Fill `secrets.github_webhook` and `secrets.github_api_token`. GitHub Actions PAT secret must not be named `GITHUB_` — use `OPSLAKE_GITHUB_PAT`. If `jira` is in `sources`, copy skill 106 Jira flow keys* (project keys, started_statuses, jira views) — otherwise ignore skill Jira Flow Extract. 6. Canonical paths in the template (raw JSON vs curated Hive Parquet). Two buckets, always. Keep `streamlit/data/contract_views.yaml` in sync with `views` so mart tabs skip correctly. ## Done when - Starter kit cloned (not opsdl). - `docs/lake-scope-contract.yaml` is filled and committed. - `views` is the complete catalog surface later activities will create. - `ci` is XOR; `lake_repo` and `oidc_sub` are set. ## Feeds Activity Model the Data Lake as CDK Stacks.
2	Model the Data Lake as CDK Stacks	MCE-2	Purpose Model every AWS resource the lake needs as a Python CDK app (TypeScript only …	Read the 'MCE-2 Model the Data Lake as CDK Stacks' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Model every AWS resource the lake needs as a Python CDK app (TypeScript only if the host repo is already TypeScript). No console clicks. No separate SAM template. ## Inputs - Starter clone `infra/` already has four stacks. Customize from the filled contract; do not scaffold empty. - `docs/lake-scope-contract.yaml`. Instantiate only sources/tables listed there. - Skill AWS CDK Infrastructure as Code → Recipe / Copy-this tests. ## Steps 1. Open skill AWS CDK Infrastructure as Code. Files to touch: `infra/ops_lake/scope.py` (loader), then `deploy_stack.py`, `ingestion_stack.py`, `catalog_stack.py`, `eda_stack.py` only as the contract requires. Do not scaffold empty `infra/`. 2. `DeployStack`: GitHub (or GitLab) OIDC + deploy IAM role. Trust both name-only and ID-qualified `sub` patterns (skill GitHub Actions CI/CD → Copy-this IAM trust). Output `DeployRoleArn` for `AWS_DEPLOY_ROLE_ARN`. 3. `IngestionStack`: raw + curated + athena-results buckets (names from contract), `BucketEncryption.KMS_MANAGED` and `bucket_key_enabled=True`, lifecycle; HTTP API; one webhook Lambda per scoped source; `GithubBackfillFn` (10 min / 1024 MB); EventBridge poller for Projects v2 if github scoped (heartbeat until GraphQL ships); DynamoDB cursor; SQS; `ParquetTransformFn`. Do not attach a customer-managed `kms.Key` to buckets that notify SQS. If jira scoped, wire `jira_flow.py` copied from `infra/lambdas/jira_flow.py.example`. 4. `CatalogStack`: Glue database `athena_database`; one crawler per contract `views` on `s3://{curated}/{table}/`; Athena workgroup; named preview queries matching `views`. 5. `EdaStack`: Streamlit on ECS Fargate + ALB (`linux/amd64`). Dockerfile COPY `.py`, `pages/`, and `data/`. Grant task role Athena, Glue GetTables, curated-bucket read, Athena-results read/write. Output `StreamlitMartUrl`. 6. Environment from CDK context / SSM. Synth: from repo root `cdk synth -c skip_bundling=true` (or `cd infra` if that is how this clone is wired). 7. Pytest: `tests/unit/test_deploy_stack.py`, `test_ingestion_stack.py`, `test_catalog_stack.py`, `test_eda_stack.py`, `test_story_data.py`. Assert BucketKeyEnabled on all three lake buckets. ## Done when - `infra/app.py` instantiates all four stacks. - Synth tests pass, including S3 Bucket Keys. - GitHub-only contract → no GitlabWebhookFn / live JiraPollFn. ## Feeds XOR: Deploy via GitHub Actions* or Deploy via GitLab CI.
3	Deploy via GitHub Actions	MCE-3	Purpose Deploy the CDK app through GitHub Actions with OIDC. Production path for a GitHub-hosted …	Read the 'MCE-3 Deploy via GitHub Actions' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Deploy the CDK app through GitHub Actions with OIDC. Production path for a GitHub-hosted lake repo. ## Inputs - Lake Scope Contract (`ci` must be `github_actions`). - CDK Application (includes DeployStack). - Starter `.github/workflows/deploy.yml` — copy from the clone; do not invent a second pipeline. Template: `https://github.com/FeatureFactory-io/ops-lake-starter/blob/main/.github/workflows/deploy.yml`. - Skill GitHub Actions CI/CD → Recipe. ## Skip when `ci` is `gitlab_ci`. Do not create deploy.yml "just in case". ## Steps 1. Keep starter `.github/workflows/deploy.yml`. 2. Chicken-and-egg: laptop, same region as the contract: `cdk bootstrap` then `cdk deploy OpsLakeDeploy` (or `--all`) once. Read `DeployRoleArn` from stack outputs. GitHub → Settings → Secrets and variables → Actions → New repository secret `AWS_DEPLOY_ROLE_ARN`. Also `OPSLAKE_GITHUB_PAT` (`repo`, `read:org`). Never `GITHUB_`. After that, never `cdk deploy` a shared env from a laptop. 3. `permissions: id-token: write`. Assume the deploy role via `aws-actions/configure-aws-credentials`. No `AWS_ACCESS_KEY_ID`. 4. Jobs: pytest → `cdk deploy --all --require-approval never`. Print `DeployRoleArn`, `GithubWebhookUrl`, `StreamlitMartUrl` to the job summary. IAM simulate Streamlit task-role S3 read. 5. OIDC trust both* `repo:{org}/{lake_repo}:` and `repo:{org}@/{lake_repo}@:`. A `main` push AssumeRoleWithWebIdentity must succeed (check the Actions log). 6. After first GHA deploy, record URLs/ARNs in `docs/deploy-manifest.yaml`. Org webhook registration is the next human step (activity Stand Up Serverless Ingestion) — deploy does not by itself land events. ## Done when - Workflow file is the only shared-env deploy path. - OIDC covers both `sub` shapes; `AWS_DEPLOY_ROLE_ARN` is set. ## Feeds Pipeline + populate-lake. Shared-env resources appear only after this workflow runs on `main`.
3	Deploy via GitLab CI	MCE-3	Purpose Deploy the CDK app through GitLab CI with OIDC. Alternative to GitHub Actions. Inputs …	Read the 'MCE-3 Deploy via GitLab CI' activity from playbook 'Delivery Constraint & Opportunity Analysis' (workflow 'Model and Deploy Infrastructure as Code'). Follow its guidance when assisting me. ## Purpose Deploy the CDK app through GitLab CI with OIDC. Alternative to GitHub Actions. ## Inputs - Lake Scope Contract (`ci` must be `gitlab_ci`). - CDK Application. - Starter `.gitlab-ci.yml.example` — copy to `.gitlab-ci.yml`. Skill GitLab CI/CD → Recipe. ## Skip when `ci` is `github_actions`. Running both against one AWS account causes split-brain deploys. ## Steps 1. Copy `.gitlab-ci.yml.example` to `.gitlab-ci.yml`. Do not also keep a live GitHub `deploy.yml` against this account. 2. Chicken-and-egg: laptop `cdk bootstrap` then one `cdk deploy OpsLakeDeploy` to mint the GitLab OIDC IAM role. Store the role ARN as CI/CD variable `AWS_DEPLOY_ROLE_ARN` (not `AWS_ACCESS_KEY_ID`). Trust `aud` = `https://gitlab.com` (or this instance URL) and the project `sub`. 3. `id_tokens` block mints the OIDC token. pytest → `cdk synth` → `cdk diff` → `cdk deploy` with `when: manual` for production. Register GitLab Environment `production`. 4. After first deploy, record URLs/ARNs in `docs/deploy-manifest.yaml`. ## Done when - `.gitlab-ci.yml` is the only shared-env deploy path. - GitHub Actions deploy workflow is absent. - A `main` pipeline assumed the role. ## Feeds Pipeline (populate equivalent on GitLab if you add it; otherwise GitHub populate-lake is the GitHub-ci path).

Activities in Model and Deploy Infrastructure as Code