Sign in to create and edit playbooks. Sign In Register

Populate-Lake Pipeline

Document Required

Description

GitHub Actions populate-lake.yml: sync PAT, async GithubBackfillFn, sync_raw_to_curated.py, Glue crawlers, Athena LIMIT 10 proof. Not laptop scripts.

Metadata
Type:
Document
Required:
Yes
Created:
Sep 15, 2026
Updated:
Sep 15, 2026
Producer Activity
Backfill Historical GitHub Events

Build the Operational Data Lake Pipeline

Consumer Activities 4
  • Store Compressed and Partitioned in S3

    Build the Operational Data Lake Pipeline Optional

  • Look until Trusted

    Constraint and Flow Analysis Required

  • Catalog with AWS Glue

    Build the Operational Data Lake Pipeline Required

  • Present the Catalog via Athena

    Build the Operational Data Lake Pipeline Required