Partition Keys Stay Out of Parquet Body
partition-keys-out-of-parquet-body · always_apply: yes
Content
Hive partition keys (source, dt) live in the S3 path {table}/source={source}/dt={yyyy-mm-dd}/. Do not also write those columns into the Parquet body. Glue crawlers then see duplicate columns and cataloging fails. Template: infra/lambdas/parquet_transform.py _normalize_row omits source.
Details
- Slug:
partition-keys-out-of-parquet-body- Always apply:
- Yes
- Created:
- Sep 15, 2026
- Updated:
- Sep 15, 2026
Playbook
Activities 1
- Store Compressed and Partitioned in S3 Build the Operational Data Lake Pipeline