Skill: S3 Data Partitioning and Compression

Capability Domain: Data Engineering
Technology Stack: Amazon S3, Apache Parquet
Linked Activities: Store Compressed and Partitioned in S3

Canonical layout

• Raw: s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json
• Curated: s3://{curated}/{table}/source={source}/dt={yyyy-mm-dd}/part-{uuid}.parquet
• {table} ∈ Scope Contract views. Two buckets. Never a single s3://ops-lake/source=gitlab/dt= layout.

Recipe

1. ParquetTransformFn on ObjectCreated under raw prefix. Write Parquet with pyarrow bundled in the Lambda asset — not awswrangler (layer size / synth Docker). Unit synth uses CDK context skip_bundling.
2. Map GitHub event types onto contract tables (push → github_commits, pull_request → github_pull_requests, issues → github_issues, workflow_run → github_workflow_runs; backfill kind commits/pulls/issues/workflow_runs). Projects v2 via GraphQL poller → github_project_items when that poller is real.
3. Omit partition keys from the Parquet body. source and dt live only in the S3 path. Duplicating source in the file makes Glue report duplicate columns.
4. SQS buffer for webhooks. After backfill, do not wait on SQS — scripts/sync_raw_to_curated.py batch-invokes ParquetTransformFn.
5. If compression <5x vs JSON, stop and fix schema.
6. Lake buckets are SSE-KMS with S3 Bucket Keys (bucket_key_enabled=True in IngestionStack). Glue crawlers and Athena GetObject every Parquet part; without Bucket Keys those reads are KMS Decrypt per file.

Pitfalls

• One Parquet file per webhook event (webhook path).
• SSE-KMS without S3 Bucket Keys — one-file-per-event plus crawler/Athena reads burns the 20k KMS Free Tier in hours.
• source both as Hive partition and Parquet column.
• Partitioning finer than dt without a query need.