Research

MachinoAI Research

IMPORTANTAI Agents

Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

AgentActionBench evaluates whether research agents actually observe, plan, implement, execute, and verify experiments instead of only producing plausible final repositories.

Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua LinSep 10, 2026NLPCC Shared Task / arXiv preprint11 min read
AI AgentsAgent EvaluationReproducibilityMCPBenchmarkAI4ScienceLLM Agentsprocess-oriented evaluationaction tracesrubric-based evaluation

TL;DR

The benchmark contains 150 papers, including 120 ML and 30 AI4Science papers, with more than 10,000 rubric items; the best shared-task system scores 49.64% overall.

Why It Matters

This is directly relevant to production research agents because it turns hidden execution behaviour into auditable traces and separates planning/code-generation capability from actual runtime reliability.

Research Brief

The shortest useful explanation.

The benchmark contains 150 papers, including 120 ML and 30 AI4Science papers, with more than 10,000 rubric items; the best shared-task system scores 49.64% overall.

process-oriented evaluationaction tracesrubric-based evaluationexperiment reproductionexecution reliabilityAgent EvaluationReproducibilityMCP

Core Explanation

We introduce AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains. An MCP-based Action Recorder captures agent behaviour throughout reproduction and evaluates the resulting traces with paper-specific rubrics.

Why It Matters

This is directly relevant to production research agents because it turns hidden execution behaviour into auditable traces and separates planning/code-generation capability from actual runtime reliability.

AgentActionBench framework connecting a paper to an agent, standardised tools, recorded actions, rubrics, evaluator and reliable results.

Figure 2 — AgentActionBench framework

Original paper figure showing the standardised execution environment, recorded actions, rubrics, evaluator and reliable-results output.

Section 01

Problem

01 The Problem

Research-agent evaluations can score final repositories or descriptions without proving that the agent actually read the source, implemented the required method, executed the code and reproduced the result. AgentActionBench targets this observability gap by evaluating the reproduction process itself across ML and AI4Science papers.

1Final-artifact scoring can accept plausible files without execution evidence.2Reproduction requires comprehension, planning, implementation, execution and verification.3AI4Science adds domain diversity beyond standard ML.

Section 02

How It Works

03 How It Works

A paper is paired with a paper-specific rubric. The agent operates inside a standardised workspace through Read, Write and Execute MCP tools. Each invocation becomes a chronological structured action log. For each rubric item, an evaluator extracts relevant logs and assigns Pass or Fail; the overall score is the importance-weighted proportion of passed rubric items.

1Read captures inspection.2Write captures planning and implementation.3Execute captures runtime behaviour.4Result Matching uses Read and Execute evidence.

Section 03

Architecture

05 Architecture

The system has a source paper, agent, standardised execution environment and evaluation pipeline. The Action Recorder mediates Read/Write/Execute calls and emits a chronological JSON trace. Rubrics consume the trace and an LLM evaluator produces Pass/Fail judgments.

1MCP is the action boundary.2Execution is isolated for safety and fairness.3The agent itself need not be modified.

Section 04

Experiments

06 Experiments

AgentActionBench contains 150 papers: 120 ML and 30 AI4Science. The human subset contains 12 ML and 3 AI4Science papers and averages 67.75 rubric items per paper. Model-assisted expansion produces more than 10,000 rubric items with an average of 69.92 per paper. The leaderboard reports YNU-HPCC-Task11-AgentRep at 49.64% overall, zzunlp_wu at 24.70%, Codex-GPT-5.4 at 24.19% and QueenAgent at 7.03%. For the best team, rubric-type scores are 43.73% Paper Observation, 82.45% Plan Writing, 61.89% Code Implementation, 18.04% Command Execution and 17.24% Result Matching. Model-generated versus human rubrics show Pearson 0.93 ± 0.04 and Spearman 0.88 ± 0.07.

1Models: Claude Code scaffold + Claude-Sonnet for rubric generation; ChatGPT-4o-mini evaluator; GPT-5.4 + Codex reproduction baseline.2Hardware: two NVIDIA L40S GPUs.3Primary question: whether process-aware scoring exposes runtime failures hidden by final artifacts.

Formulas

Rubric item

ri=(ci,si,ti)r_i=(c_i,s_i,t_i)

A rubric item contains a criterion, importance score and reproduction stage.

c_icriterion
s_iimportance
t_istage

Reproduction score

mathrmscore=fracsumi[Ji=mathrmPass]sisumisi\\mathrm{score}=\\frac{\\sum_i [J_i=\\mathrm{Pass}]s_i}{\\sum_i s_i}

The benchmark score is the importance-weighted proportion of passed rubric items.

J_iPass/Fail judgment
s_iimportance

Related Research

AI Agents

Generative Agents

An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.

AI Agents

MIRA

MIRA is an autonomous medical agent evaluated in a sandboxed EHR workflow with tools for diagnosis, testing, treatment, and admission decisions.

AI Agents

Agentic Reasoning

A tool-using agent framework that extends LLM reasoning with web search, coding, and structured reasoning memory for deep research tasks.

References

Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

arXiv:2504.17192

Prior paper-to-code reproduction work.

HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

arXiv:2604.17745

Related hierarchical multi-agent reproduction framework.

arXiv abstractarXiv HTML v1