Backfill Historical GitHub Events

BPE-2 Order: #2 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Guidance

Purpose

Pull historical GitHub issues, PRs, commits, and workflow runs into the raw bucket so Constraint and Flow Analysis can compute cycle time. Webhooks only catch events after they are registered.

Inputs

Lake Scope Contract (repos, secrets.github_api_token).
CDK Application (GithubBackfillFn already declared).
Starter workflow .github/workflows/populate-lake.yml — template https://github.com/FeatureFactory-io/ops-lake-starter/blob/main/.github/workflows/populate-lake.yml. Do not laptop-invoke against shared env.
Skills Serverless Data Ingestion and Populate the Lake.

Steps

Keep starter infra/lambdas/github_backfill.py and populate-lake.yml. Open skill Populate the Lake.
Issues and pulls: state=all and per_page=100. GitHub's default is open only — that was the empty cycle-time dashboard. Wrap each item {backfill: true, kind, repo, payload} with kind in issues | pulls | commits | workflow_runs.
Paginate Link: rel=next for list endpoints. Commits API pagination is not the same as issue/PR Link headers — follow page= until empty. Timeout 10 minutes / 1024 MB.
Repo secret OPSLAKE_GITHUB_PAT (repo, read:org) — never GITHUB_*. Populate-lake syncs it into Secrets Manager.
GitHub → Actions → Populate the Lake → Run workflow. Watch CloudWatch for written=N. Never laptop invoke against shared env.
After raw objects land, the same workflow runs scripts/sync_raw_to_curated.py then crawlers then Athena proof. Athena LIMIT 10 is activity Present the Catalog via Athena; here you only need written=N and raw keys under s3://{raw}/github/.

Done when

Glue github_issues will contain closed rows after Parquet + crawl (payload.state='closed'), not only open.
CloudWatch shows written=N for issues, pulls, commits, workflow_runs.

Feeds

Store Compressed and Partitioned in S3, then Glue/Athena. Cycle-time EDA depends on this history.

Honesty

GithubProjectsPollFn is a heartbeat until GraphQL pagination ships. Do not join github_project_items as if Master Roadmap items are populated.

Details

Order:: #2
Phase:
Predecessor:: BPE-1 Stand Up Serverless Ingestion from Scoped Sources
Successor:: BPE-3 Store Compressed and Partitioned in S3
Created:: Sep 15, 2026
Last Updated:: Sep 17, 2026

Workflow

Build the Operational Data Lake Pipeline

Pull operational history from the systems people actually work in into the lake, including closed work, so duration and wait …

View Workflow

Assigned Agent

Data Platform Engineer

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Required Skills

Populate the Lake Data Engineering GitHub Actions, Lambda, Glue, Athena
Serverless Data Ingestion Data Engineering AWS CDK, Lambda, EventBridge, API Gateway, Kinesis Firehose

Rules

Infrastructure as Code First infrastructure-as-code-first
Least-Privilege IAM and Encryption by Default least-privilege-iam-and-encryption-by-default
Never Name GitHub Actions Secrets GITHUB_* never-name-gha-secrets-github-prefix
No EMR or Persistent Clusters for Ingestion no-emr-or-persistent-clusters-for-ingestion

Input Artifacts 2

CDK Application (Infrastructure Stacks) Document Required
Produced by: Model the Data Lake as CDK Stacks
Lake Scope Contract Document Required
Produced by: Scope Sources, Org, and Environment

Output Artifacts 1

Populate-Lake Pipeline Document Required