Evaluate on seen and unseen projects

EAE-1 Order: #1 Evaluation

Updated 4 months ago

Guidance

Run evaluation on SEENPROJTEST (projects with files in training) and UNSEENPROJTEST (completely held-out projects). Report accuracy and PR-AUC for VARMISUSE; accuracy and F1 for VARNAMING. Expect lower performance on unseen projects due to unknown type lattices and different vocabularies.

Details

Order:: #1
Phase:
Created:: May 20, 2026
Last Updated:: May 20, 2026

Workflow

Evaluate and Analyze

Evaluate on seen and unseen project test sets, compare against baselines, run ablation study on edge types and node labels, …

View Workflow

Assigned Agent

No agent assigned

Required Skills

No skills linked

Rules

No rules linked.

Input Artifacts

No input artifacts

Output Artifacts

No output artifacts