Explore Data and Find Signal - EDA with Python, Pandas and Jupyter

Description

Go from raw selected datasets to validated conclusions: organize and clean the data, engineer a feature table aimed at one or more candidate target variables (Y), and quantify how much information those features actually carry about Y before trusting any finding.

Activities in this Phase

Order	Name	Actions
Organize and Clean Data
#1	Load selected datasets into DataFrames	Read the 'ODA-1 Load selected datasets into DataFrames' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Read each selected catalog dataset into a pandas Series/DataFrame using the appropriate reader (read_csv with explicit sep/header/index_col/na_values, read_json, read_fwf, or chunked read_csv(chunksize=) for very large extracts). Repair obviously sparse/irregular files (supply explicit `names=`, then `ffill()` a sparse category column before `set_index()`).
#2	Inspect structure	Read the 'ODA-2 Inspect structure' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Immediately after loading, check `.head()`/`.tail()`, `len()`, `.shape`, `.dtypes`, `.index`, `.columns` (and `.ndim` for any raw numpy arrays). This is the standard first look that catches wrong delimiters, misparsed headers, or unexpected types before you build anything on top of them.
#3	Handle missing and invalid data	Read the 'ODA-3 Handle missing and invalid data' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Detect gaps with `isnull()`/`notnull()`. Then choose deletion (`dropna()`, whole rows/columns only), imputation (constant/mean via masked assignment, or `fillna(method="ffill"/"bfill")`), selective replacement (`replace()`), or backfill from a secondary source (`combine_first()`) — the right choice depends on domain knowledge about what a gap means for that variable, not a default rule.
#4	Combine and deduplicate datasets	Read the 'ODA-4 Combine and deduplicate datasets' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. If two selected datasets are complementary (different variables, shared key), `pd.merge()`/`.join()` them; if they're similar-schema (same variables, e.g. across time periods or regions), `pd.concat()` them. After any vertical concat, check for and resolve duplicate index labels or rows (`duplicated()`, `drop_duplicates()`).
#5	Reshape data for the framing questions	Read the 'ODA-5 Reshape data for the framing questions' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Organize and Clean Data'). Follow its guidance when assisting me. Convert between long ("tidy") and wide ("square") layouts with `stack()`/`unstack()`/`pivot()`, and build a hierarchical (multi-index) structure if your questions need more than one categorical grouping key (e.g. State + Year). Choose the shape each later step actually needs rather than leaving everything in whatever shape it was loaded in.
Explore and Engineer Features
#1	Compute descriptive statistics per candidate variable	Read the 'EFS-1 Compute descriptive statistics per candidate variable' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Explore and Engineer Features'). Follow its guidance when assisting me. For every candidate variable, compute sum/mean/median/std/count/min/max (with `skipna`), sort/rank it, and look at cumulative/period-over-period measures (`cumsum`, `diff`) and uniqueness/value_counts for categoricals. This is the "smell and sense of the data" step before you commit to any Y or feature.
#2	Choose candidate target variable(s) Y	Read the 'EFS-2 Choose candidate target variable(s) Y' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Explore and Engineer Features'). Follow its guidance when assisting me. From the Framing Brief's questions, pick the concrete column(s) that would answer them if you understood what drove them — this is Y. Note whether Y is continuous (→ regression), categorical/binary (→ classification), or whether there is no single Y and the task is genuinely unsupervised (→ clustering, handled in ACE).
#3	Engineer and derive features	Read the 'EFS-3 Engineer and derive features' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Explore and Engineer Features'). Follow its guidance when assisting me. Build the feature table: arithmetic transforms across columns (index-aligned), `groupby()` aggregation to compare Y across categorical breakdowns, `cut()`/`qcut()` to discretize a continuous variable, `get_dummies()` to encode categoricals for models that need numeric input, and `series.map(func)` for one-off custom transforms.
#4	Cross-tabulate categorical candidates against Y	Read the 'EFS-4 Cross-tabulate categorical candidates against Y' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Explore and Engineer Features'). Follow its guidance when assisting me. Run `pd.crosstab(feature, Y, margins=True)` for each categorical feature candidate as a cheap first-pass independence check before committing to the heavier correlation/modeling work in the next workflow.
#5	Compute correlation/covariance and significance between features and Y	Read the 'EFS-5 Compute correlation/covariance and significance between features and Y' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Explore and Engineer Features'). Follow its guidance when assisting me. Compute pairwise correlation/covariance (`frame.corr()`, `.cov()`, `.corrwith(Y)`) across the feature table, then get significance via `scipy.stats.pearsonr()` since pandas doesn't supply p-values — align mismatched indexes manually first if needed. This produces the ranked candidate feature–Y relationships that Phase 2's ACE workflow will validate.
Assess Information Gain and Conclude
#1	Interpret correlation with p-value and rule out spurious relationships	Read the 'ACE-1 Interpret correlation with p-value and rule out spurious relationships' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Assess Information Gain and Conclude'). Follow its guidance when assisting me. Cross each correlation's sign/magnitude against its p-value (p≤.01 vs not) before treating it as evidence. Remember correlation is not causation — a real correlation may share a hidden confounder — and a non-significant linear correlation doesn't rule out a non-linear relationship.
#2	Design and run a predictive experiment	Read the 'ACE-2 Design and run a predictive experiment' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Assess Information Gain and Conclude'). Follow its guidance when assisting me. Split the feature table 70:30 train/test (seed the RNG first). Choose the model by Y's type (per the Target Variable (Y) Definition from EFS): continuous Y → OLS `LinearRegression`, or `Ridge` if predictors are collinear; categorical/binary Y → `LogisticRegression` (use `cut()` first to quantize a continuous proxy into labels if needed, and set `C` well above the default 1.0, e.g. ~20). Fit on training data only, engineering nonlinear predictors (squares/logs/interaction terms) first if a plain linear fit under-captures curvature, then apply the fitted model to the held-out test set. Never touch the test data while fitting.
#3	Evaluate model quality	Read the 'ACE-3 Evaluate model quality' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Assess Information Gain and Conclude'). Follow its guidance when assisting me. Build a confusion matrix (classification) or compare predicted-vs-actual on train and test sets (regression); derive accuracy/precision/sensitivity/specificity and choose which one matters for the question at hand. Compare train-set fit against test-set fit to judge overfitting before trusting the model.
#4	Use clustering as an alternative when correlation is weak	Read the 'ACE-4 Use clustering as an alternative when correlation is weak' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Assess Information Gain and Conclude'). Follow its guidance when assisting me. When linear correlation is weak or absent but you suspect structure, scale the features (`sklearn.preprocessing.scale()`), run k-means (`KMeans(n_clusters=k)`, k chosen by judgment), and assign human-meaningful labels to the resulting clusters — cluster IDs alone mean nothing.
#5	Rank features by information gain and draw conclusions	Read the 'ACE-5 Rank features by information gain and draw conclusions' activity from playbook 'EDA with Python, Pandas and Jupyter' (workflow 'Assess Information Gain and Conclude'). Follow its guidance when assisting me. Fit a Random Forest Regressor on the feature table against Y to get a feature-importance ranking (an ensemble-of-trees estimate of how much each feature actually informs Y), cross-check it against the regression coefficients and correlations from earlier steps, and write the surviving, validated findings as the phase's Conclusions.