Skill: AWS Glue Cataloging

Capability Domain: Data Engineering
Technology Stack: AWS Glue, Glue Crawlers, Hive Metastore
Linked Activities: Catalog with AWS Glue

Recipe

1. One database: Scope Contract athena_database (default ops_catalog).
2. One crawler per views entry, target s3://{curated}/{table}/ (partition subdirs source=). Names like {table}_crawler.
3. Schedule: daily EventBridge cron(0 6 * * ? *) plus a one-shot start from populate-lake after Parquet sync. Do not trigger a crawler per S3 object.
4. Built-in Parquet classifier. Custom classifier only if auto-detect fails.
5. Define crawlers in CatalogStack. No console-created crawlers.
6. After populate-lake, wait until each crawler State=READY, then Athena proof.

Escalate

Schema-drift that changes an existing column type — Operations Analyst before apply.

Pitfalls

• Crawlers for gitlab_* / jira_* when those views are not in the contract.
• Per-object crawler triggers (cost + race with incomplete Parquet).
• Duplicate source columns from Parquet body + partition (fix in ParquetTransformFn first).