Evaluation - Program Graphs for Code Analysis

Description

Validate model performance against baselines and ablations, and demonstrate practical relevance through real-world bug discovery.

Activities in this Phase

Order	Name	Actions
Evaluate and Analyze
#1	Discover real-world bugs	Read the 'EAE-1 Discover real-world bugs' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Rank variable usage locations by model confidence that a different variable should be used. Inspect the top-N locations (top 500 in the paper) where the model strongly disagrees with the ground truth. Classify findings as true bugs vs. valid alternatives (e.g. aliases). Report confirmed bugs with code snippets.
#1	Run ablation study	Read the 'EAE-1 Run ablation study' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Systematically vary: (1) edge types used — syntax only, semantic only, combined; (2) node label representation — subtokens vs full tokens vs disabled. Measure impact on both VARNAMING and VARMISUSE accuracy to attribute performance gains to specific design choices.
#1	Compare against baselines	Read the 'EAE-1 Compare against baselines' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Compare GGNN against: LOC (local bidirectional GRU, no usage context), AVGBIRNN (bidirectional RNN averaged over all usages). For VARNAMING also compare against AVGLBL (log-bilinear model over 4-token context windows). All baselines use same variable selection objective: argmax of c(t)^T u(t,v).
#1	Evaluate on seen and unseen projects	Read the 'EAE-1 Evaluate on seen and unseen projects' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Evaluate and Analyze'). Follow its guidance when assisting me. Run evaluation on SEENPROJTEST (projects with files in training) and UNSEENPROJTEST (completely held-out projects). Report accuracy and PR-AUC for VARMISUSE; accuracy and F1 for VARNAMING. Expect lower performance on unseen projects due to unknown type lattices and different vocabularies.