OpenDiscoveryTrace Dataset Released to Evaluate AI Scientist Agent Trajectories and Reasoning Processes

Traditional benchmarks for autonomous AI scientists focus almost exclusively on final outputs, such as completed code, formulated hypotheses, or draft papers. This output-centric approach makes it difficult to audit the underlying scientific methodology, diagnose intermediate failure modes, or distinguish actual systematic reasoning from lucky guesses. To address this gap, researchers have introduced OpenDiscoveryTrace, a public dataset containing complete trajectories that capture the step-by-step reasoning of AI agents. The dataset contains 558 trajectories recording structured, nine-field-per-step traces, including model thoughts, tool invocations, observations, errors, and self-reported confidence. These traces represent AI models executing 124 distinct scientific tasks across domains like drug discovery, genomics, and materials science. It covers several frontier models, such as GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro, alongside multiple open-weight models. Initial pilot analyses demonstrate that process traces expose critical behavioral differences that output-only metrics fail to capture. For instance, while the studied frontier models achieved comparable final success rates of 84 percent to 89 percent, their internal execution paths differed drastically. Claude Opus 4.6 generated thirty times more internal errors per trajectory than GPT-5.4, highlighting the utility of trace auditing for safer and more reliable AI agent development.