Combine and deduplicate datasets

ODA-4 Order: #4 Explore Data and Find Signal Has Dependencies

Updated 1 month, 2 weeks ago

Guidance

If two selected datasets are complementary (different variables, shared key), pd.merge()/.join() them; if they're similar-schema (same variables, e.g. across time periods or regions), pd.concat() them. After any vertical concat, check for and resolve duplicate index labels or rows (duplicated(), drop_duplicates()).

Details

Order:: #4
Phase:
Predecessor:: ODA-3 Handle missing and invalid data
Successor:: ODA-5 Reshape data for the framing questions
Created:: Sep 18, 2026
Last Updated:: Sep 18, 2026

Workflow

Organize and Clean Data

Turn the selected raw datasets into one clean, combined, correctly-shaped working DataFrame — the shared substrate every later exploration step …

View Workflow

Assigned Agent

EDA Practitioner

The person or AI system running this playbook end to end — analyst, programmer, and …

Required Skills

Data Combination (Merge/Join/Concat) Data Integration pandas

Rules

Merge for complementary data, concat for similar data merge-for-complementary-data-concat-for-similar-data
Watch for duplicate keys after vertical concat watch-for-duplicate-keys-after-vertical-concat

Input Artifacts

No input artifacts

Output Artifacts

No output artifacts