Data Preparation - Program Graphs for Code Analysis

Description

Prepare a richly annotated program graph dataset ready for model training — from raw repositories through compiled projects, graph construction, and node representation computation.

Activities in this Phase

Order	Name	Actions
Collect and Compile Source
#1	Split dataset by file	Read the 'CSE-1 Split dataset by file' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Collect and Compile Source'). Follow its guidance when assisting me. Partition into train/validation/test splits along source file boundaries — all examples from one file must go to the same split. Reserve held-out projects (no files in training) for UNSEENPROJTEST evaluation. Suggested split: 60-10-30.
#1	Scrape target repositories
#1	Extract variable usage slots	Read the 'CSE-1 Extract variable usage slots' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Collect and Compile Source'). Follow its guidance when assisting me. For each variable usage location (excluding declarations), check that at least one other type-compatible variable is in scope. Each such location becomes a training slot. Record the ground-truth variable at each slot.
#1	Filter compilable projects
Build Program Graphs
#1	Parse source to AST	Read the 'BGS-1 Parse source to AST' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Build Program Graphs'). Follow its guidance when assisting me. Parse each source file to its Abstract Syntax Tree. Label syntax nodes with nonterminal names and syntax tokens with their string representation. Add Child edges connecting parent-child AST relationships and NextToken edges connecting consecutive tokens.
#1	Add data flow edges	Read the 'BGS-1 Add data flow edges' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Build Program Graphs'). Follow its guidance when assisting me. For each variable token v, compute DR(v) (last read locations) and DW(v) (last write locations). Add LastRead and LastWrite edges accordingly — these may span non-sequential locations including across loop boundaries. Add ComputedFrom edges connecting assignment targets to all variable tokens in the right-hand side expression.
#1	Add semantic shortcut edges	Read the 'BGS-1 Add semantic shortcut edges' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Build Program Graphs'). Follow its guidance when assisting me. Add LastLexicalUse edges chaining all occurrences of the same variable (independent of data flow). Add ReturnsTo edges connecting return tokens to their method declaration. Add FormalArgName edges connecting call-site arguments to the formal parameter names they map to.
#1	Add backward edges	Read the 'BGS-1 Add backward edges' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Build Program Graphs'). Follow its guidance when assisting me. For every edge type added (Child, NextToken, LastRead, LastWrite, ComputedFrom, LastLexicalUse, ReturnsTo, FormalArgName), add the corresponding backward edge by transposing the adjacency matrix. This doubles edge count and types, aids information propagation, and increases model expressiveness.
Compute Node Representations
#1	Compute subtoken embeddings	Read the 'CRS-1 Compute subtoken embeddings' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Compute Node Representations'). Follow its guidance when assisting me. Split each node name on camelCase and pascal_case boundaries into subtokens (e.g. classTypes → [class, types]). Learn an embedding for each subtoken. Compute the node name embedding as the average of its subtoken embeddings.
#1	Assemble initial node state
#1	Compute type hierarchy embeddings	Read the 'CRS-1 Compute type hierarchy embeddings' activity from playbook 'Program Graphs for Code Analysis' (workflow 'Compute Node Representations'). Follow its guidance when assisting me. For each variable, collect the full set of its supertypes (type lattice τ(v)). Learn an embedding r(τ) for each known type; use "UNKTYPE" for unseen types. Compute the variable's type representation as the element-wise maximum over {r(τ) : τ ∈ τ(v)}. During training, randomly sample a non-empty subset of τ*(v) as regularization.