Store Compressed and Partitioned in S3

BPE-3 Order: #3 Datalake Formation Has Dependencies

Updated 1 month, 2 weeks ago

Purpose

Turn raw landing-zone JSON into Hive-partitioned Parquet in the curated bucket.

Raw Operational Events (S3 Landing Zone).
CDK Application (ParquetTransformFn lives in IngestionStack).
Lake Scope Contract paths: raw vs curated.
Populate-Lake Pipeline (batch transform after backfill).
Skill S3 Data Partitioning and Compression → Recipe step 2 (event→table map).

Raw: s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json
Curated: s3://{curated}/{table}/source={source}/dt={yyyy-mm-dd}/part-{uuid}.parquet
{table} is a name from contract views. Two buckets, always.

Open skill S3 Data Partitioning and Compression. Handler path: infra/lambdas/parquet_transform.py (_normalize_row omits source).
Event → table map (skill 64 recipe 2): push → github_commits, pull_request → github_pull_requests, issues → github_issues, workflow_run → github_workflow_runs; backfill kind commits/pulls/issues/workflow_runs. Projects v2 → github_project_items only when that poller is real.
ParquetTransformFn writes pyarrow Parquet. Omit source/dt from the row body — they are Hive partitions only.
Webhooks: SQS buffer. Backfill: scripts/sync_raw_to_curated.py from populate-lake only, not a laptop against shared env.
Lifecycle: hot recent partitions Standard; cold raw/history IA or Glacier.
Confirm lake buckets have S3 Bucket Keys (bucket_key_enabled=True) via pytest synth (tests/unit/test_ingestion_stack.py), not the S3 console encryption tab alone.
If jira is scoped, run Backfill Historical Jira Issues and Status History (sibling of GitHub backfill) before Glue — this activity's predecessor is GitHub backfill only because workflows cannot cross-link both.

Catalog with AWS Glue.

Pull operational history from the systems people actually work in into the lake, including closed work, so duration and wait …

Agent: Data Platform Engineer Role Owns Datalake Formation: Scope Contract → Python CDK (DeployStack + …

Infrastructure as Code First infrastructure-as-code-first
Least-Privilege IAM and Encryption by Default least-privilege-iam-and-encryption-by-default
No EMR or Persistent Clusters for Ingestion no-emr-or-persistent-clusters-for-ingestion
Parquet Over Raw Formats parquet-over-raw-formats
Partition by Source and Date at Minimum partition-by-source-and-date-at-minimum
Partition Keys Stay Out of Parquet Body partition-keys-out-of-parquet-body
S3 Bucket Keys With SSE-KMS s3-bucket-keys-with-sse-kms

CDK Application (Infrastructure Stacks) Document Required
Produced by: Model the Data Lake as CDK Stacks
Jira Flow Extract Contract Document
Produced by: Backfill Historical Jira Issues and Status History
Lake Scope Contract Document Required
Produced by: Scope Sources, Org, and Environment
Populate-Lake Pipeline Document
Produced by: Backfill Historical GitHub Events
Raw Operational Events (S3 Landing Zone) Document Required
Produced by: Stand Up Serverless Ingestion from Scoped Sources