Activities - Evaluate and Analyze

#	Name	Abbrev.	Guidance	Actions
Evaluation 4
1	Compare against baselines	EAE-1	Compare GGNN against: LOC (local bidirectional GRU, no usage context), AVGBIRNN (bidirectional RNN averaged over …	Read the 'EAE-1 Compare against baselines' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Compare GGNN against: LOC (local bidirectional GRU, no usage context), AVGBIRNN (bidirectional RNN averaged over all usages). For VARNAMING also compare against AVGLBL (log-bilinear model over 4-token context windows). All baselines use same variable selection objective: argmax of c(t)^T u(t,v).
1	Discover real-world bugs	EAE-1	Rank variable usage locations by model confidence that a different variable should be used. Inspect …	Read the 'EAE-1 Discover real-world bugs' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Rank variable usage locations by model confidence that a different variable should be used. Inspect the top-N locations (top 500 in the paper) where the model strongly disagrees with the ground truth. Classify findings as true bugs vs. valid alternatives (e.g. aliases). Report confirmed bugs with code snippets.
1	Evaluate on seen and unseen projects	EAE-1	Run evaluation on SEENPROJTEST (projects with files in training) and UNSEENPROJTEST (completely held-out projects). Report …	Read the 'EAE-1 Evaluate on seen and unseen projects' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Run evaluation on SEENPROJTEST (projects with files in training) and UNSEENPROJTEST (completely held-out projects). Report accuracy and PR-AUC for VARMISUSE; accuracy and F1 for VARNAMING. Expect lower performance on unseen projects due to unknown type lattices and different vocabularies.
1	Run ablation study	EAE-1	Systematically vary: (1) edge types used — syntax only, semantic only, combined; (2) node label …	Read the 'EAE-1 Run ablation study' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Systematically vary: (1) edge types used — syntax only, semantic only, combined; (2) node label representation — subtokens vs full tokens vs disabled. Measure impact on both VARNAMING and VARMISUSE accuracy to attribute performance gains to specific design choices.

Activities in Evaluate and Analyze