Sign in to create and edit playbooks. Sign In Register

Scope Sources, Org, and Environment

MCE-1 Order: #1 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Guidance

Purpose

Write the Lake Scope Contract before any AWS resource is modeled. Later activities read only this file for org, sources, CI host, AWS account, region, buckets, Athena view names, lake repo, OIDC sub, and secret keys.

Inputs

  • Artifact Operations Lake Starter Kit — clone https://github.com/FeatureFactory-io/ops-lake-starter (Use this template). Not the live opsdl lake.
  • Human target (GitHub org / GitLab group / Jira site).
  • aws sts get-caller-identity and configured region.
  • Where the lake repo will live (GitHub vs GitLab).
  • Skill Scope the Operations Lake → Copy-this YAML (and Jira flow keys if jira is scoped).

Steps

  1. Clone the starter kit into the org (GitHub → Use this template). Rename lake_repo if the clone is not ops-lake-starter. Open skill Scope the Operations Lake.
  2. Fill docs/lake-scope-contract.yaml in the clone: org, host, sources. List repos: gh api orgs/{org}/repos --paginate. Projects v2: gh api graphql -f query='query($org:String!){organization(login:$org){projectsV2(first:20){nodes{id title}}}}' -f org=YOUR_ORG. Default github entities = commits, pull_requests, issues, workflow_runs, project_items. Drop unused. Do not add gitlab or jira unless this org uses them.
  3. Set ci to exactly one of github_actions | gitlab_ci matching host.
  4. Fill aws_account, aws_region, env, lake_name, lake_repo, oidc_sub. Derive views as {source}_{entity} (copy-this list is in the YAML).
  5. Fill secrets.github_webhook and secrets.github_api_token. GitHub Actions PAT secret must not be named GITHUB_* — use OPSLAKE_GITHUB_PAT. If jira is in sources, copy skill 106 Jira flow keys (project keys, started_statuses, jira views) — otherwise ignore skill Jira Flow Extract.
  6. Canonical paths in the template (raw JSON vs curated Hive Parquet). Two buckets, always. Keep streamlit/data/contract_views.yaml in sync with views so mart tabs skip correctly.

Done when

  • Starter kit cloned (not opsdl).
  • docs/lake-scope-contract.yaml is filled and committed.
  • views is the complete catalog surface later activities will create.
  • ci is XOR; lake_repo and oidc_sub are set.

Feeds

Activity Model the Data Lake as CDK Stacks.

Details
Order:
#1
Phase:
Successor:
MCE-2 Model the Data Lake as CDK Stacks
Created:
Sep 14, 2026
Last Updated:
Sep 17, 2026
Workflow
Model and Deploy Infrastructure as Code

Lock the lake's scope and stand the AWS estate up as code so every later step lands on the same …

View Workflow
Assigned Agent
Data Platform Engineer

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Required Skills
Rules
Input Artifacts 1
Output Artifacts 1