Skill: Serverless Data Ingestion

Capability Domain: Data Engineering
Technology Stack: AWS CDK, Lambda, EventBridge, API Gateway
Linked Activities: Stand Up Serverless Ingestion from Scoped Sources, Backfill Historical GitHub Events, Backfill Historical Jira Issues and Status History

Recipe

1. Read Scope Contract sources. Implement one Lambda per listed source. Do not implement the others.
2. GitHub (if scoped): GithubWebhookFn on HTTP API /webhooks/github. Verify X-Hub-Signature-256 against Secrets Manager ops-lake/github-webhook-secret. Return 200; SQS for slow writes.
3. GitHub history (required for cycle time): GithubBackfillFn — issues and pulls with state=all, per_page=100, paginate Link headers. Wrap {backfill, kind, repo, payload}. Template: infra/lambdas/github_backfill.py. Timeout 10 min / 1024 MB. Invoke async from populate-lake.
4. GitLab (if scoped): GitlabWebhookFn on /webhooks/gitlab, X-Gitlab-Token vs ops-lake/gitlab-webhook-secret.
5. Jira (if scoped): JiraPollFn EventBridge rate(5 minutes), cursor in DynamoDB ops-lake-ingestion-cursors PK source. This poller is not the lead/cycle extract. Historical issues, changelog status segments, parameterized Started set, and business-day lead/cycle live in skill Jira Flow Extract and activity Backfill Historical Jira Issues and Status History. Implement that path whenever jira is in sources and STEP 3/4 will score issue flow time.
6. Write unmodified JSON to s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json. Parquet is the next activity.
7. GithubProjectsPollFn is a heartbeat until GraphQL pagination ships. Do not treat github_project_items as a populated Projects v2 feed.

Copy-this: infra/lambdas/github_webhook.py

Handler: verify HMAC, parse GitHub event, put JSON to raw bucket with key {source}/yyyy/mm/dd/{delivery-id}.json, return 200. Dedupe downstream on event id; do not reject redeliveries.

Escalate

• Payload > API Gateway 10MB.
• Jira rate-limit twice in one day → raise interval, do not swallow.

Pitfalls

• Creating Jira/GitLab functions for a GitHub-only contract.
• Fetching GitHub issues without state=all (default is open-only).
• Treating a Jira current-status poll as cycle time (no changelog timeline, no resolutiondate).