Research

Evaluation and Benchmarking of LLM Agents: A Survey

Mahmoud Mohammadi12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on reliability, safety and capability evaluation.

Updated on Oct 3, 2026

01

Abstract

  • This survey focuses specifically on how large language model agents should be evaluated and benchmarked.
  • It treats an LLM agent as a system that uses an LLM together with tools, memory, planning, and environment interaction.
  • Agent evaluation is broader than evaluating a standalone language model on static question-answering.
  • The survey organizes existing evaluation work by capabilities, environments, benchmarks, metrics, and evaluation methods.
  • It highlights the rapid growth of agent benchmarks and the difficulty of comparing results across heterogeneous settings.
  • The work is a survey and taxonomy rather than a new agent architecture.
  • Its central goal is to identify what current agent evaluations measure and what they still miss.

02

Introduction

  • Standalone LLM benchmarks generally evaluate language understanding or generation from fixed inputs.
  • Agents operate over multiple steps and can change the environment.
  • Agent performance therefore depends on planning, tool use, memory, decision making, and recovery.
  • A final answer can be correct even when the trajectory contains inefficient or unsafe behavior.
  • Conversely, an agent can make an early mistake and recover successfully.
  • Evaluation must therefore consider both outcomes and interaction trajectories.
  • The survey argues for evaluation protocols that reflect these system-level properties.

03

Problem

  • The core problem is obtaining reliable, comparable measurements of LLM-agent capability.
  • Agents must be evaluated on task success as well as intermediate behaviors.
  • Long-horizon tasks create compounding-error effects that are invisible in single-turn tests.
  • Tool use introduces additional failure modes such as invalid calls and incorrect arguments.
  • Memory introduces persistence, retrieval, and stale-information problems.
  • Dynamic environments make evaluation dependent on state and interaction history.
  • The survey therefore calls for multidimensional agent evaluation rather than one aggregate score.

04

Methodology

  • An agent receives a task and repeatedly observes its state, reasons, and takes actions.
  • The evaluator records actions, observations, intermediate states, and final outcomes.
  • Automatic evaluators can compare final environment states against task-specific success criteria.
  • Execution-based evaluation is especially useful for coding and tool-use tasks.
  • Human evaluation can assess qualities that are difficult to formalize, such as helpfulness or interaction quality.
  • LLM-as-a-judge can provide scalable evaluation but introduces evaluator bias and reliability concerns.
  • Robust evaluation should distinguish agent capability from evaluator artifacts.
Figure 1

05

Architecture

  • Agent evaluation contains multiple layers: model capability, agent policy, tools, environment, and evaluator.
  • Reasoning quality concerns whether the agent derives appropriate intermediate decisions.
  • Planning quality concerns decomposition and ordering of actions.
  • Tool-use quality concerns selecting valid tools and arguments.
  • Memory quality concerns storing, retrieving, and using relevant information.
  • Interaction quality concerns following protocols and responding appropriately to environment feedback.
  • Final task success measures whether the desired objective was achieved.
  • Efficiency can be measured through steps, tool calls, latency, or computational cost.

06

Dataset

  • The survey covers a broad set of agent benchmark environments.
  • Software-engineering benchmarks evaluate repository-level issue resolution.
  • Web benchmarks evaluate navigation, information retrieval, and state-changing web tasks.
  • Game and simulated environments test sequential decision making.
  • Household and embodied environments test grounded interaction.
  • Knowledge and database environments test structured reasoning and querying.
  • Multi-domain benchmarks test whether agents generalize across qualitatively different tasks.
  • Representative benchmark families discussed in the literature include AgentBench, WebArena, SWE-bench, ALFWorld, and related environments.

07

Training

  • Evaluation commonly uses pretrained LLMs as the reasoning backbone rather than training a new foundation model for each benchmark.
  • Different model families can be compared under the same environment and task protocol.
  • Agent scaffolds can add memory, planning, retrieval, reflection, or tool-use mechanisms.
  • Tool-enabled evaluation tests capabilities that are unavailable in pure text generation.
  • Model size and instruction tuning can strongly affect agent behavior.
  • The survey emphasizes separating backbone-model capability from agent-system improvements.
  • Controlled comparisons are necessary to determine whether gains come from the model, prompt, tools, or orchestration.

08

Experiments

  • Task success rate is one of the most common agent-level metrics.
  • Other metrics include exact-match outcomes, execution-based pass rates, reward, trajectory length, and tool-call efficiency.
  • Human judgments can assess answer quality, usefulness, or interaction behavior.
  • Safety-oriented evaluation can measure policy violations or harmful actions.
  • Robustness evaluation tests performance under perturbations or changing environment states.
  • Cost-aware evaluation can track inference calls, tokens, latency, and tool-execution overhead.
  • A single success score can hide substantial differences in efficiency, reliability, and safety.

09

Baselines

  • Benchmark comparisons must control for environment, task distribution, model backbone, prompting, tools, and evaluation protocol.
  • AgentBench provides multi-environment evaluation across different interaction settings.
  • WebArena emphasizes realistic web interaction and long-horizon tasks.
  • SWE-bench evaluates real software-engineering issue resolution using tests.
  • ALFWorld evaluates language agents in interactive household environments.
  • Different benchmarks expose different weaknesses, so no single benchmark captures all agent capabilities.
  • Cross-benchmark comparisons should therefore be interpreted cautiously.

10

Results

  • The survey finds that current benchmarks increasingly evaluate realistic multi-step tasks rather than static questions.
  • Agent evaluation reveals weaknesses that are not visible on conventional LLM benchmarks.
  • Long-horizon planning remains difficult.
  • Tool use can fail even when the underlying language model knows the required information.
  • Memory and state tracking remain important bottlenecks.
  • Stronger language models generally improve agent performance but do not eliminate system-level failures.
  • Benchmark realism increases the importance of environment grounding and reliable execution.

11

Conclusion

  • The survey concludes that LLM-agent evaluation must be multidimensional and environment-aware.
  • Final task success remains important but should be complemented by trajectory, efficiency, safety, and robustness measures.
  • Realistic interactive benchmarks are increasingly important for measuring autonomous behavior.
  • Evaluation should isolate contributions from models, prompts, tools, memory, and orchestration.
  • Standardized protocols are needed to improve reproducibility and comparability.
  • Future benchmarks should cover longer horizons, dynamic environments, safety, personalization, and multi-agent interaction.
  • Better evaluators are needed for both automatic and human-centered properties.
  • The broader direction is toward rigorous system-level evaluation rather than treating agents as ordinary text-generation models.

12

References

  • The survey is “Evaluation and Benchmarking of LLM Agents: A Survey,” arXiv:2507.21504.
  • The paper surveys evaluation methods and benchmarks for LLM-based agents.
  • Primary source: https://arxiv.org/abs/2507.21504.
  • It distinguishes agent evaluation from conventional LLM evaluation.
  • Major evaluation dimensions include reasoning, planning, tool use, memory, interaction, task success, efficiency, robustness, and safety.
  • Representative benchmarks discussed include AgentBench, WebArena, SWE-bench, and ALFWorld.
  • The survey's central message is that agent evaluation requires realistic environments, multidimensional metrics, and controlled system-level comparisons.

Continue reading

Related research

Browse all research