Model the Data Lake as CDK Stacks

MCE-2 Order: #2 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Guidance

Purpose

Model every AWS resource the lake needs as a Python CDK app (TypeScript only if the host repo is already TypeScript). No console clicks. No separate SAM template.

Inputs

Starter clone infra/ already has four stacks. Customize from the filled contract; do not scaffold empty.
docs/lake-scope-contract.yaml. Instantiate only sources/tables listed there.
Skill AWS CDK Infrastructure as Code → Recipe / Copy-this tests.

Steps

Open skill AWS CDK Infrastructure as Code. Files to touch: infra/ops_lake/scope.py (loader), then deploy_stack.py, ingestion_stack.py, catalog_stack.py, eda_stack.py only as the contract requires. Do not scaffold empty infra/.
DeployStack: GitHub (or GitLab) OIDC + deploy IAM role. Trust both name-only and ID-qualified sub patterns (skill GitHub Actions CI/CD → Copy-this IAM trust). Output DeployRoleArn for AWS_DEPLOY_ROLE_ARN.
IngestionStack: raw + curated + athena-results buckets (names from contract), BucketEncryption.KMS_MANAGED and bucket_key_enabled=True, lifecycle; HTTP API; one webhook Lambda per scoped source; GithubBackfillFn (10 min / 1024 MB); EventBridge poller for Projects v2 if github scoped (heartbeat until GraphQL ships); DynamoDB cursor; SQS; ParquetTransformFn. Do not attach a customer-managed kms.Key to buckets that notify SQS. If jira scoped, wire jira_flow.py copied from infra/lambdas/jira_flow.py.example.
CatalogStack: Glue database athena_database; one crawler per contract views on s3://{curated}/{table}/; Athena workgroup; named preview queries matching views.
EdaStack: Streamlit on ECS Fargate + ALB (linux/amd64). Dockerfile COPY *.py, pages/, and data/. Grant task role Athena, Glue GetTables, curated-bucket read, Athena-results read/write. Output StreamlitMartUrl.
Environment from CDK context / SSM. Synth: from repo root cdk synth -c skip_bundling=true (or cd infra if that is how this clone is wired).
Pytest: tests/unit/test_deploy_stack.py, test_ingestion_stack.py, test_catalog_stack.py, test_eda_stack.py, test_story_data.py. Assert BucketKeyEnabled on all three lake buckets.

Done when

infra/app.py instantiates all four stacks.
Synth tests pass, including S3 Bucket Keys.
GitHub-only contract → no GitlabWebhookFn / live JiraPollFn.

Feeds

XOR: Deploy via GitHub Actions or Deploy via GitLab CI.

Details

Order:: #2
Phase:
Predecessor:: MCE-1 Scope Sources, Org, and Environment
Successor:: MCE-3 Deploy via GitLab CI
Created:: Aug 31, 2026
Last Updated:: Sep 17, 2026

Workflow

Model and Deploy Infrastructure as Code

Lock the lake's scope and stand the AWS estate up as code so every later step lands on the same …

View Workflow

Assigned Agent

Data Platform Engineer

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Required Skills

AWS CDK Infrastructure as Code Infrastructure as Code AWS CDK (Python default), CloudFormation

Rules

Execute by Workflow Order execute-by-step-labels
GitHub OIDC Trust Both Sub Shapes github-oidc-trust-both-sub-shapes
Infrastructure as Code First infrastructure-as-code-first
Least-Privilege IAM and Encryption by Default least-privilege-iam-and-encryption-by-default
No EMR or Persistent Clusters for Ingestion no-emr-or-persistent-clusters-for-ingestion
S3 Bucket Keys With SSE-KMS s3-bucket-keys-with-sse-kms
Streamlit Mart on ECS ALB Never App Runner streamlit-mart-ecs-alb-not-apprunner

Input Artifacts 1

Lake Scope Contract Document Required
Produced by: Scope Sources, Org, and Environment

Output Artifacts 5

CDK Application (Infrastructure Stacks) Document Required
OIDC Deploy Role (GitHub Actions) Document Required
Operations Lake Starter Kit Document Required
Playbook Recapture Ledger Document
Streamlit Data Mart Document Required