No EMR or Persistent Clusters for Ingestion
no-emr-or-persistent-clusters-for-ingestion · always_apply: yes
Content
Keep the ingestion path serverless and simple: CDK-modeled Lambda functions (and a CDK-modeled Kinesis Firehose delivery stream for streams) only. Do not stand up EMR or any persistent cluster to get operational data from GitLab/GitHub/Jira into the lake — that complexity belongs, if ever, to the broader analytics portfolio, not to basic ingestion.
Details
- Slug:
no-emr-or-persistent-clusters-for-ingestion- Always apply:
- Yes
- Created:
- Aug 31, 2026
- Updated:
- Aug 31, 2026
Playbook
Activities 4
- Stand Up Serverless Ingestion from Scoped Sources Build the Operational Data Lake Pipeline
- Backfill Historical GitHub Events Build the Operational Data Lake Pipeline
- Store Compressed and Partitioned in S3 Build the Operational Data Lake Pipeline
- Model the Data Lake as CDK Stacks Model and Deploy Infrastructure as Code