Present the Catalog via Athena

BPE-5 Order: #5 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Guidance

Purpose

Prove the Athena surface. Glue crawler tables are the contract views. Downstream consumers (Streamlit, Pandas, MCP) query those tables (or thin views if you add them), never random Glue names.

Inputs

AWS Glue Data Catalog.
Lake Scope Contract (views, athena_workgroup default ops-lake-analysts, athena_database default ops_catalog).
Populate-Lake Pipeline.
Skills Athena SQL Querying and Populate the Lake.
SQL template sql/proof_views.sql (placeholders {database}, {view}).

Steps

Open skills Athena SQL Querying and Populate the Lake.
One workgroup from contract athena_workgroup (AWS name default ops-lake-analysts) with EnforceWorkGroupConfiguration and a per-query scan cap (start 5GB).
After crawlers READY, populate-lake runs scripts/create_athena_views.py — LIMIT 10 per contract view; skip missing tables. Do not laptop-run against shared env. Console alternative: Athena query editor → workgroup dropdown ops-lake-analysts → database ops_catalog.
Copy-this proof: SELECT COUNT(*) AS n FROM {view} WHERE source = '{source}' then SELECT * FROM {view} LIMIT 10. Backfill JSON: extract $.payload.*. Prefer closed snapshots when deduping issue numbers. Record each QueryExecutionId on artifact Athena Catalog Views.
Glue tables already are the views; the script is a proof runner, not a second catalog of DDL.
Hands off to Operations Analyst when all contract views that have data are queryable.

Done when

Each populated contract view runs a SELECT ... LIMIT 10 in the workgroup.
Streamlit/Pandas/MCP are pointed at these tables/views only.

Feeds

Constraint and Flow Analysis (artifact gate).

Details

Order:: #5
Phase:
Predecessor:: BPE-4 Catalog with AWS Glue
Created:: Aug 31, 2026
Last Updated:: Sep 17, 2026

Workflow

Build the Operational Data Lake Pipeline

Pull operational history from the systems people actually work in into the lake, including closed work, so duration and wait …

View Workflow

Assigned Agent

Data Platform Engineer

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Required Skills

Athena SQL Querying Analytics Amazon Athena
Populate the Lake Data Engineering GitHub Actions, Lambda, Glue, Athena

Rules

Infrastructure as Code First infrastructure-as-code-first
Least-Privilege IAM and Encryption by Default least-privilege-iam-and-encryption-by-default

Input Artifacts 4

AWS Glue Data Catalog Document Required
Produced by: Catalog with AWS Glue
CDK Application (Infrastructure Stacks) Document Required
Produced by: Model the Data Lake as CDK Stacks
Lake Scope Contract Document Required
Produced by: Scope Sources, Org, and Environment
Populate-Lake Pipeline Document Required
Produced by: Backfill Historical GitHub Events

Output Artifacts 1

Athena Catalog Views Document Required