Build the Operational Data Lake Pipeline

BPE #2 Delivery Constraint & Opportunity Analysis

Description

Pull operational history from the systems people actually work in into the lake, including closed work, so duration and wait are facts rather than estimates. When jira is scoped, persist issue snapshots plus a status timeline and compute lead/cycle from that extract (skill Jira Flow Extract) — a current-status poller is not enough.

Activities Flow (6)
cluster_datalake_formation Datalake Formation activity_243 BPE1 Stand Up Serverless Ingestion from Scoped Sources activity_393 BPE2 Backfill Historical GitHub Events activity_243->activity_393 activity_399 BPE3 Backfill Historical Jira Issues and Status History activity_243->activity_399 activity_244 BPE3 Store Compressed and Partitioned in S3 activity_393->activity_244 activity_245 BPE4 Catalog with AWS Glue activity_244->activity_245 activity_246 BPE5 Present the Catalog via Athena activity_245->activity_246