Is AI’s scientific research capability overestimated?
Cutting-edge Coding Agents are translating machine learning research into automatically executable workflows, but most existing studies focus on well-structured codebases, single tasks or the field of computer science. When it comes to scientific research projects scattered across different disciplines that lack robust training and evaluation processes, there is still a lack of systematic measurement of whether Agents can truly improve models. Testing Agents directly on original scientific research codebases is also unreliable. When the original repository lacks clear metric definitions or standardized data splits, Agents may exploit evaluation loopholes to achieve superficial improvements rather than actually enhancing model performance.
To address this, the Anthropic team proposes Agent4Sci, a long-range automated climbing evaluation framework for ML for Science tasks, covering 11 scientific domains, 20 benchmarks and 25 tasks.
Paper link: https://arxiv.org/abs/2610.28763
In the experiment, four cutting-edge Agents collectively improved all 20 benchmarks, with Claude Opus achieving an average relative improvement of 10.4%. However, by analyzing the operation trajectories of the Agents, the research team found that their improvements mainly came from conventional machine learning engineering tricks rather than research-level insights. After adding three plugin skills, Opus’s average relative improvement further rose to 20.1%. This work was selected for the Evaluation and Datasets track at NeurIPS 2026.
The construction of Agent4Sci is divided into two steps: first, standardize scientific research tasks, and then use a three-stage protocol to isolate climbing, evaluation and auditing, limiting the space for cheating exposed in pilot experiments.
The first step is four-stage screening plus manual standardization. The research team screened benchmarks through four stages: domain classification, automatic screening, manual verification and in-depth review. Taking the NeurIPS 2025 Data and Benchmarks track as an example, the research team finally selected 10 benchmarks from 497 candidate papers, with a rejection rate of about 98%. Each selected task must pass the baseline reproduction of the paper, and complete the standardization of data splitting, operating environment and training process.
The second step is the three-stage evaluation protocol. First, the climbing Agent obtains papers, code, training data and validation data, modifies the code, trains the model and submits results within a fixed resource budget, but has no access to the test set. Subsequently, an independent evaluation Agent uses a standardized evaluation process to calculate the final metrics on the held-out test set. Finally, the auditing Agent checks the operation logs and submitted outputs to identify evaluation speculation and other academic misconduct.
Figure | Agent4Sci evaluation protocol, where the three Agents for climbing, evaluation and auditing are mutually isolated
This design specifically addresses the evaluation loopholes exposed in pilot experiments. The research team further used auditing Agents from three model families, Claude, Codex and Gemini, to cross-verify part of the operation trajectories, and the auditing conclusions remained consistent, with no runs flagged. However, independent evaluation and auditing do not mean that all covert speculative behaviors can be ruled out.
The experiment compared Claude Opus 4.6, Claude Sonnet 4.6, Gemini 3.1 Pro and Codex (GPT-5.4). All four ran through their respective official CLIs at the highest inference intensity. Each experiment was provided with 8 H200 GPUs and a 12-hour budget, with no human involvement in specific decisions during the experiment. The average relative improvements of the four Agents were 10.4%, 9.6%, 6.8% and 4.4% respectively. Not every Agent achieved improvements on all benchmarks, but all 20 benchmarks were improved by at least one Agent, with individual improvement rates ranging from 1.1% to 64.4%.
Figure | Average relative improvements of four cutting-edge Agents on Agent4Sci
Specific cases show that the improvements can come from both conventional model optimization and more targeted task adaptation. In the PFDelta power flow estimation task, Opus reduced the average power balance loss by 64.4% by adjusting auxiliary losses, increasing the weight of the mean squared error loss, and using cosine annealing with warm-up. In the cone detection task of Mars-Bench, Opus replaced the original RetinaNet with Faster R-CNN which is more suitable for the task, increasing the mAP from 16.5 to 30.4.
However, trajectory analysis reveals the limitations behind these gains. Agents extensively adopt mature methods such as learning rate adjustment, model deepening and widening, backbone network replacement and weight decay, while interventions targeting specific scientific problems are relatively rare. There are also obvious differences in the working methods of different Agents: Opus continuously monitors logs and iterates experiments, Codex tends to submit long-running scripts early and then waits for execution to finish, while Gemini falls in between the two.
Aiming at the recurring problems in the trajectories, the research team designed three lightweight plugin skills for research ideation, experiment management and timed checklists respectively. The plugin skills nearly doubled the average improvement of the two Agents. Opus improved 18 out of 20 benchmarks, with PFDelta jumping from 64.4% to 91.0%, MassSpecGym reaching 39.5%, and CarbonGlobe reaching 35.3%. The new solution on PFDelta reads more like physical reasoning: the Agent identified that the power balance loss can be derived from Kirchhoff’s law and does not require labels, reconstructed it as the main training loss, used GNN predictions to provide a warm start for the AC power flow equation during inference, and gradually refined the voltage variables to push the physical residual towards zero.
Figure | Comparison of improvement rates on each benchmark between default runs and runs with added skills
The research team stated that although the performance improvements mainly come from machine learning engineering rather than scientific reasoning, Agent4Sci still translates the model improvement capability in automated scientific research into a reproducible and auditable evaluation problem. It not only measures whether Agents can improve metrics, but also tries to distinguish whether the improvements come from mature engineering tricks or the understanding of the scientific problem itself.
In the future, whether Coding Agents can move from executing experiments more efficiently to proposing more valuable research ideas will become an important issue for measuring automated scientific research capabilities.
For more technical details, please refer to the original paper.
This article is from the WeChat Official Account “Academic Headlines” (ID: SciTouTiao) authorized for release by 36Kr
