Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
AgentActionBench evaluates whether research agents actually observe, plan, implement, execute, and verify experiments instead of only producing plausible final repositories.
Key takeaway
The benchmark contains 150 papers, including 120 ML and 30 AI4Science papers, with more than 10,000 rubric items; the best shared-task system scores 49.64% overall.
Updated on Oct 2, 2026
01
Abstract
AgentActionBench evaluates scientific experiment reproduction through the agent’s observable actions rather than only the final repository. It records reading, writing, execution, and result-matching behavior against paper-specific rubrics.
02
Introduction
Research-agent evaluations can score final artifacts without proving that the agent actually read the source, implemented the required method, executed the code, and reproduced the result.
03
Problem
The missing measurement is process-level evidence: whether the agent performed the actions necessary for faithful reproduction, not merely whether its final files look plausible.
04
Background
The benchmark connects scientific-paper reproduction with trace-based agent evaluation, MCP tool use, reproducibility benchmarks, and process-aware grading.
05
Methodology
Each paper is paired with a paper-specific rubric. Agents operate through Read, Write, and Execute MCP tools, producing chronological structured traces. Evaluators map trace evidence to Pass/Fail rubric items and compute an importance-weighted score.
Figure 2 — AgentActionBench framework
Original paper figure showing the standardised execution environment, recorded actions, rubrics, evaluator and reliable-results output.
arXiv HTML v1
Formula notes
Mathematical details
Rubric item
A rubric item contains a criterion, importance score and reproduction stage.
- c_i
- -criterion
- s_i
- -importance
- t_i
- -stage
Reproduction score
The benchmark score is the importance-weighted proportion of passed rubric items.
- J_i
- -Pass/Fail judgment
- s_i
- -importance
06
Architecture
The system contains a source paper, agent, standardized workspace, Action Recorder, structured trace, paper-specific rubric, and evaluator. The recorder mediates tool actions and emits JSON events.
07
Dataset
AgentActionBench contains 150 papers: 120 ML and 30 AI4Science. The human subset contains 12 ML and 3 AI4Science papers and averages 67.75 rubric items per paper. Model-assisted expansion produces more than 10,000 rubric items with an average of 69.92 per paper.
08
Training
The benchmark evaluates agent behavior rather than introducing a new foundation-model training procedure. Model-assisted rubric expansion is used to scale evaluation criteria.
09
Experiments
AgentActionBench contains 150 papers. The leaderboard reports 49.64% for YNU-HPCC-Task11-AgentRep, 24.70% for zzunlp_wu, 24.19% for Codex-GPT-5.4, and 7.03% for QueenAgent. The best team scores 82.45% on Plan Writing but 18.04% on Command Execution and 17.24% on Result Matching.
10
Baselines
Participating agent systems form the benchmark comparison set, with human-authored rubrics serving as an annotation reference for model-generated rubric quality.
11
Results
12
Ablation
The evaluation dimensions are Paper Observation, Plan Writing, Code Implementation, Command Execution, and Result Matching.
13
Limitations
Trace completeness, rubric quality, evaluator judgments, and AI4Science coverage limit interpretation.
14
Conclusion
AgentActionBench shifts experiment-reproduction evaluation toward observable process evidence, complementing final-artifact grading.
15
References
AgentActionBench/NLPCC 2026 Shared Task 11 primary paper and related scientific-agent reproduction benchmarks.
Source trail
Primary papers, implementation pages, and external source records used to ground this explainer.
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
arXiv:2504.17192
Prior paper-to-code reproduction work.
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
arXiv:2604.17745
Related hierarchical multi-agent reproduction framework.
01arXiv abstract02arXiv HTML v1Continue reading
Related research
GitSwarm
GitSwarm turns inference-time work into a persistent, branchable body of reusable contributions that later agents can build on.
AI AgentsLongHarness
LongHarness helps agents reason over 100M-token document collections by keeping evidence and conclusions as explicit, revisable states that guide one another.
AI AgentsGenerative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.