Stand Up Serverless Ingestion from Scoped Sources

Purpose

Land raw JSON events from scoped sources into the raw bucket. Implement only Lambdas the Scope Contract lists. Org webhook registration is a human GitHub (or GitLab) Settings step after the first CI deploy — deploy does not by itself land events.

Inputs

• Lake Scope Contract (sources, views, bucket names, secrets.github_webhook).
• CDK Application (IngestionStack already declares the functions). Gate is the artifact — predecessors cannot cross workflows.
• Stack output GithubWebhookUrl (job summary after deploy.yml, also docs/deploy-manifest.yaml).
• Skill Serverless Data Ingestion → Recipe.

Steps

1. Open skill Serverless Data Ingestion. Handlers already live under infra/lambdas/ in the starter (github_webhook.py, and jira_flow.py only after you copy infra/lambdas/jira_flow.py.example). Edit signatures / secret names from the contract; do not scaffold empty.
2. Confirm Secrets Manager ops-lake/github-webhook-secret exists (CDK creates it; populate-lake may sync). Value must match the GitHub webhook secret you will paste in org Settings.
3. Register the org webhook (human): GitHub org → Settings → Webhooks → Add webhook. Payload URL = GithubWebhookUrl. Content type application/json. Secret = the Secrets Manager value. SSL verify. Events: Issues, Pull requests, Pushes, Workflow runs. Save. Send a test ping.
4. GitHub handler: GithubWebhookFn behind /webhooks/github, verify X-Hub-Signature-256. Return 200; enqueue SQS for slow work. Prove: CloudWatch /aws/lambda/GithubWebhookFn; raw object at s3://{raw}/github/{yyyy}/{mm}/{dd}/.
5. GitLab: implement only if gitlab is in the contract. Secret ops-lake/gitlab-webhook-secret; header X-Gitlab-Token (skill 63).
6. Jira: implement only if jira is in the contract. Copy infra/lambdas/jira_flow.py.example → jira_flow.py. The poller lands raw JSON. Historical issues, changelog, and lead/cycle clocks are Backfill Historical Jira Issues and Status History plus skill Jira Flow Extract — a current-status poller is not a cycle-time extract.
7. Write unmodified JSON to s3://{raw}/{source}/{yyyy}/{mm}/{dd}/{event-id}.json. No Parquet here.
8. Idempotent on event id. GithubProjectsPollFn remains a heartbeat until GraphQL pagination ships — do not join github_project_items as a populated Master Roadmap.

Done when

• Org webhook is registered (or GitLab equivalent if that host is scoped).
• One working handler per scoped source; zero handlers for unscoped systems.
• Test ping produced a raw object (or CloudWatch shows a verified 200).
• Secrets live in Secrets Manager / SSM, referenced from CDK.

Feeds

Backfill Historical GitHub Events (when github is scoped), Backfill Historical Jira Issues and Status History (when jira is scoped), then Store Compressed and Partitioned in S3.