Skill: Populate the Lake

Capability Domain: Data Engineering
Technology Stack: GitHub Actions, Lambda, Glue, Athena
Linked Activities: Backfill Historical GitHub Events, Catalog with AWS Glue, Present the Catalog via Athena

When to use

After STEP 1 GHA has deployed stacks. Webhooks only catch the future. Constraint analysis and cycle-time EDA need closed/merged history.

Recipe

1. Repo secret OPSLAKE_GITHUB_PAT (repo, read:org). Never GITHUB_*.
2. Workflow .github/workflows/populate-lake.yml (workflow_dispatch + daily cron). Template: starter kit .github/workflows/populate-lake.yml (https://github.com/FeatureFactory-io/ops-lake-starter/blob/main/.github/workflows/populate-lake.yml).
3. Job steps (in order):
   - Sync PAT into Secrets Manager ops-lake/github-api-token.
   - Optionally reset Glue tables + curated prefixes when re-crawling a broken catalog.
   - Invoke GithubBackfillFn with --invocation-type Event (async). Sync CLI invoke times out ~3 minutes; the function needs up to 10 minutes.
   - Poll CloudWatch /aws/lambda/GithubBackfillFn for GithubBackfillFn written=N with N>0.
   - Run scripts/sync_raw_to_curated.py to batch-invoke ParquetTransformFn (do not wait on SQS).
   - Start each {view}_crawler; wait until State=READY.
   - scripts/create_athena_views.py — LIMIT 10 proof per contract view; skip views with no Glue table.
4. Do not run these scripts from a laptop against the shared account.

Pitfalls

• GitHub issues/pulls API defaults to state=open. Without state=all the lake has zero closed issues and cycle time is empty.
• Waiting on SQS after a bulk backfill — invoke the transform Lambda directly.