Scope Sources, Org, and Environment
MCE-1 Order: #1 Datalake Formation Has Dependencies
Updated 1 month, 2 weeks ago
Guidance
Purpose
Write the Lake Scope Contract before any AWS resource is modeled. Later activities read only this file for org, sources, CI host, AWS account, region, buckets, Athena view names, lake repo, OIDC sub, and secret keys.
Inputs
- Artifact Operations Lake Starter Kit — clone
https://github.com/FeatureFactory-io/ops-lake-starter(Use this template). Not the live opsdl lake. - Human target (GitHub org / GitLab group / Jira site).
aws sts get-caller-identityand configured region.- Where the lake repo will live (GitHub vs GitLab).
- Skill Scope the Operations Lake → Copy-this YAML (and Jira flow keys if jira is scoped).
Steps
- Clone the starter kit into the org (GitHub → Use this template). Rename
lake_repoif the clone is notops-lake-starter. Open skill Scope the Operations Lake. - Fill
docs/lake-scope-contract.yamlin the clone:org,host,sources. List repos:gh api orgs/{org}/repos --paginate. Projects v2:gh api graphql -f query='query($org:String!){organization(login:$org){projectsV2(first:20){nodes{id title}}}}' -f org=YOUR_ORG. Default github entities = commits, pull_requests, issues, workflow_runs, project_items. Drop unused. Do not add gitlab or jira unless this org uses them. - Set
cito exactly one ofgithub_actions|gitlab_cimatchinghost. - Fill
aws_account,aws_region,env,lake_name,lake_repo,oidc_sub. Deriveviewsas{source}_{entity}(copy-this list is in the YAML). - Fill
secrets.github_webhookandsecrets.github_api_token. GitHub Actions PAT secret must not be namedGITHUB_*— useOPSLAKE_GITHUB_PAT. Ifjirais insources, copy skill 106 Jira flow keys (project keys, started_statuses, jira views) — otherwise ignore skill Jira Flow Extract. - Canonical paths in the template (raw JSON vs curated Hive Parquet). Two buckets, always. Keep
streamlit/data/contract_views.yamlin sync withviewsso mart tabs skip correctly.
Done when
- Starter kit cloned (not opsdl).
docs/lake-scope-contract.yamlis filled and committed.viewsis the complete catalog surface later activities will create.ciis XOR;lake_repoandoidc_subare set.
Feeds
Activity Model the Data Lake as CDK Stacks.
Details
- Order:
- #1
- Phase:
- Successor:
- MCE-2 Model the Data Lake as CDK Stacks
- Created:
- Sep 14, 2026
- Last Updated:
- Sep 17, 2026
Workflow
Model and Deploy Infrastructure as Code
Lock the lake's scope and stand the AWS estate up as code so every later step lands on the same …
View WorkflowAssigned Agent
Data Platform Engineer
Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …
Required Skills
- Jira Flow Extract Data Engineering Jira REST, JQL, Changelog
- Scope the Operations Lake Discovery YAML, GitHub CLI, AWS STS
Rules
-
Execute by Workflow Order
execute-by-step-labels -
Infrastructure as Code First
infrastructure-as-code-first
Input Artifacts 1
-
Operations Lake Starter Kit
Document
Required
Produced by: Model the Data Lake as CDK Stacks
Output Artifacts 1
- Lake Scope Contract Document Required