Catalog with AWS Glue

BPE-4 Order: #4 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Purpose

Partitioned Parquet Data Lake.
Lake Scope Contract (athena_database default ops_catalog, views).
Populate-Lake Pipeline (one-shot crawler start). Starter: .github/workflows/populate-lake.yml.
Skills AWS Glue Cataloging and Populate the Lake.

Open skills AWS Glue Cataloging and Populate the Lake. Do not start crawlers in the Glue console.
One database named from contract athena_database (default ops_catalog). One crawler per views entry, scoped to s3://{curated}/{table}/. CDK already declared them — intern does not click Create crawler.
Daily schedule cron(0 6 * * ? *) plus populate-lake job step start-crawler after Parquet sync. Wait until State=READY: aws glue get-crawler --name {table}_crawler (name from CatalogStack).
Built-in Parquet classifier only unless a source fails auto-detect.
If Glue reports duplicate columns, go back to Store Compressed and Partitioned in S3 (source leaked into Parquet body). Do not patch the crawler.
Escalate to Operations Analyst before applying a crawler change that alters an existing column type.

Glue tables exist for every contract view that has Parquet (table name = view name).
No crawlers for unscoped systems.

Present the Catalog via Athena.

Pull operational history from the systems people actually work in into the lake, including closed work, so duration and wait …

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Infrastructure as Code First infrastructure-as-code-first
Partition by Source and Date at Minimum partition-by-source-and-date-at-minimum

CDK Application (Infrastructure Stacks) Document Required
Produced by: Model the Data Lake as CDK Stacks
Lake Scope Contract Document Required
Produced by: Scope Sources, Org, and Environment
Partitioned Parquet Data Lake Document Required
Produced by: Store Compressed and Partitioned in S3
Populate-Lake Pipeline Document Required
Produced by: Backfill Historical GitHub Events