Activities - Organize and Clean Data

#	Name	Abbrev.	Guidance	Actions
Explore Data and Find Signal 5
1	Load selected datasets into DataFrames	ODA-1	Read each selected catalog dataset into a pandas Series/DataFrame using the appropriate reader (read_csv with …	Read the 'ODA-1 Load selected datasets into DataFrames' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Read each selected catalog dataset into a pandas Series/DataFrame using the appropriate reader (read_csv with explicit sep/header/index_col/na_values, read_json, read_fwf, or chunked read_csv(chunksize=) for very large extracts). Repair obviously sparse/irregular files (supply explicit `names=`, then `ffill()` a sparse category column before `set_index()`).
2	Inspect structure	ODA-2	Immediately after loading, check .head()/.tail(), len(), .shape, .dtypes, .index, .columns (and .ndim for any raw …	Read the 'ODA-2 Inspect structure' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Immediately after loading, check `.head()`/`.tail()`, `len()`, `.shape`, `.dtypes`, `.index`, `.columns` (and `.ndim` for any raw numpy arrays). This is the standard first look that catches wrong delimiters, misparsed headers, or unexpected types before you build anything on top of them.
3	Handle missing and invalid data	ODA-3	Detect gaps with isnull()/notnull(). Then choose deletion (dropna(), whole rows/columns only), imputation (constant/mean via masked …	Read the 'ODA-3 Handle missing and invalid data' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Detect gaps with `isnull()`/`notnull()`. Then choose deletion (`dropna()`, whole rows/columns only), imputation (constant/mean via masked assignment, or `fillna(method="ffill"/"bfill")`), selective replacement (`replace()`), or backfill from a secondary source (`combine_first()`) — the right choice depends on domain knowledge about what a gap means for that variable, not a default rule.
4	Combine and deduplicate datasets	ODA-4	If two selected datasets are complementary (different variables, shared key), pd.merge()/.join() them; if they're similar-schema …	Read the 'ODA-4 Combine and deduplicate datasets' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. If two selected datasets are complementary (different variables, shared key), `pd.merge()`/`.join()` them; if they're similar-schema (same variables, e.g. across time periods or regions), `pd.concat()` them. After any vertical concat, check for and resolve duplicate index labels or rows (`duplicated()`, `drop_duplicates()`).
5	Reshape data for the framing questions	ODA-5	Convert between long ("tidy") and wide ("square") layouts with stack()/unstack()/pivot(), and build a hierarchical (multi-index) …	Read the 'ODA-5 Reshape data for the framing questions' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Convert between long ("tidy") and wide ("square") layouts with `stack()`/`unstack()`/`pivot()`, and build a hierarchical (multi-index) structure if your questions need more than one categorical grouping key (e.g. State + Year). Choose the shape each later step actually needs rather than leaving everything in whatever shape it was loaded in.

Activities in Organize and Clean Data