Research

The Rise and Potential of Large Language Model Based Agents: A Survey

Zhiheng Xi12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on complete agent landscape.

Updated on Oct 3, 2026

01

Abstract

  • The survey studies the rise of large language model (LLM)-based agents as systems that perceive, reason, plan, act, and learn in environments.
  • It distinguishes an LLM agent from a standalone language model by its ability to interact with external environments and use tools.
  • The survey organizes the field around agent architecture, applications, evaluation, and future directions.
  • It treats the LLM as a central controller that can be augmented with memory, planning, tools, and external environments.
  • The work is a survey and taxonomy rather than a new agent algorithm.
  • It reviews representative systems emerging around the GPT-3.5/GPT-4 generation of models.
  • The survey highlights the transition from passive text generation toward autonomous task execution.

02

Introduction

  • LLMs provide strong language understanding, reasoning, and instruction-following capabilities.
  • Agents add interaction loops, external tools, memory, planning, and environmental feedback around the model.
  • The survey identifies hallucination, limited context, knowledge cutoff, and weak long-horizon planning as important motivations for augmentation.
  • External tools can provide capabilities unavailable from the model's parameters.
  • Memory can preserve information beyond a single context window.
  • Planning decomposes complex objectives into executable steps.
  • Feedback from environments enables iterative correction.
  • These components turn an LLM from a one-shot predictor into an interactive agent.

03

Problem

  • The survey asks how LLMs can be systematically transformed into autonomous agents.
  • It identifies the need for architectures that coordinate perception, reasoning, planning, memory, and action.
  • Long-horizon tasks require state management across multiple interaction steps.
  • Agents must decide when to reason internally and when to invoke external tools.
  • Reliable execution requires handling uncertainty, errors, and changing environments.
  • Evaluation must measure more than language-generation quality.
  • The survey therefore frames LLM agents as a systems problem spanning models, tools, environments, memory, and control.

04

Methodology

  • A typical agent begins with a user goal or task.
  • The LLM interprets the goal and constructs an internal plan.
  • The agent retrieves relevant memory or external knowledge.
  • It chooses an action or tool call.
  • The environment executes the action and returns an observation.
  • The observation is incorporated into subsequent reasoning.
  • Failed actions can trigger replanning or correction.
  • The loop continues until a termination condition is reached.
  • This iterative interaction distinguishes agents from one-shot prompting.
Figure 1

An envisioned agent society

The festival scene illustrates how several agents could share an environment with people. In the kitchen, a request becomes a cooking plan with ingredients and steps. Outside, agents discuss lantern materials and use tools to calculate costs. At the concert, agents coordinate different musical roles. The user can join these activities. Together, the scenes connect task planning, tool use, and cooperation; they are an envisioned society, not evidence that a deployed system has achieved all these abilities.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

Figure 2

The perception–brain–action loop

The arrows form a feedback loop: the environment supplies observations, perception converts inputs into a usable representation, the brain chooses what to do, and action changes the environment. Inputs can include text, images, audio, and other signals. The brain combines memory and knowledge with planning and reasoning, using recall and retrieval to inform decisions. Actions can be text responses, API or tool calls, or embodied movements. In the umbrella example, the agent combines the request with weather evidence before responding or acting; the resulting state becomes the next observation.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

Figure 3

Inside the brain module

This tree breaks the brain into five capability areas: natural-language interaction, knowledge, memory, reasoning and planning, and transferability and generalization. Knowledge includes linguistic, commonsense, and actionable information, alongside challenges such as outdated facts and hallucinations. Memory methods extend context, summarize or compress experience, and retrieve relevant information. Planning includes forming a plan and reflecting on it after feedback. The branches on the right cite representative approaches; the diagram is a taxonomy of design options rather than a pipeline requiring every method.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

Figure 4

How agents perceive their environment

Perception is the interface between an environment and the agent’s decision-making system. The tree separates textual, visual, auditory, and other inputs. Visual processing includes encoders and learnable connections that align image information with a language model, with query-based and projection-based approaches shown separately. Audio can be handled through cascaded systems or methods adapted from visual processing. The cited systems illustrate different ways to make non-text observations available to the agent; a particular agent may support only some of these modalities.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

Figure 5

How agents act on the world

The action module describes how an agent produces effects beyond internal reasoning. One branch covers textual output. A second covers tools, separating learning how to use tools, invoking existing tools, and making new tools. The third covers embodied actions in physical or simulated environments. These branches distinguish saying what should happen from issuing an API call or controlling an embodied system. The listed papers are examples within each category, not successive steps that every action must follow.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

Figure 6

From individual agents to practical applications

The application taxonomy expands from one agent to interactions among agents and people. Single-agent deployment includes task-oriented work on the web or in everyday settings, innovation-oriented work, and lifecycle-oriented activity over extended interactions. Multi-agent systems may cooperate through ordered or less structured coordination, or interact adversarially. Human–agent interaction ranges from an instructor directing an executor to a partnership model, with examples in education, health, and other applications. The diagram organizes proposed and studied uses; it does not establish equal maturity or reliability across them.

Source: Zhiheng Xi et al., The Rise and Potential of Large Language Model Based Agents: A Survey (arXiv:2309.07864).

05

Architecture

  • The survey groups agent capabilities into components such as profiling, memory, planning, and action.
  • Profiling defines the agent's role, goals, capabilities, and behavioral constraints.
  • Memory can be short-term or long-term.
  • Planning can use decomposition, reflection, search, or other reasoning strategies.
  • Action connects the LLM to tools and environments.
  • Tool use can extend the model with search, calculators, code execution, APIs, and domain systems.
  • Multi-agent systems divide tasks among multiple specialized agents.
  • Communication mechanisms coordinate those agents and combine intermediate results.

06

Dataset

  • The survey reviews applications across software development, games, web interaction, robotics, scientific research, and social simulation.
  • Coding agents can inspect repositories, write code, execute tests, and repair failures.
  • Web agents can navigate pages, search information, and perform transactions.
  • Game agents can interact with simulated environments over multiple steps.
  • Embodied agents connect language reasoning to physical or simulated actions.
  • Scientific agents can retrieve and synthesize external knowledge.
  • Social simulation systems use multiple language agents to model interactions.
  • Application difficulty increases when environments are dynamic and tasks are long-horizon.

07

Training

  • The survey treats the LLM as the principal reasoning and decision-making engine.
  • Different LLMs can be used as the controller without changing the conceptual agent architecture.
  • Prompting and instruction design influence the agent's planning and action behavior.
  • External tools compensate for missing factual, computational, or environmental capabilities.
  • Retrieval systems provide external knowledge that can be updated independently of model parameters.
  • Code execution gives agents a mechanism for precise computation.
  • APIs allow agents to access specialized services.
  • The survey emphasizes that system-level augmentation can be more important than model size alone for agent capability.

08

Experiments

  • The survey discusses evaluation of LLM agents as a distinct problem from evaluating standalone LLMs.
  • Agent evaluation must account for interaction with an environment.
  • Task success is often a more meaningful metric than textual similarity.
  • Long-horizon tasks expose compounding errors that single-turn benchmarks miss.
  • Tool-use correctness and action validity are important evaluation dimensions.
  • Planning quality affects whether an agent can reach the final goal.
  • Memory quality affects performance across multiple interactions.
  • Robustness should be tested under changing states, failures, and incomplete information.
  • The survey highlights the need for standardized agent benchmarks.

09

Baselines

  • The survey compares several architectural paradigms rather than proposing a single baseline.
  • ReAct is representative of interleaved reasoning and acting.
  • Chain-of-Thought represents explicit intermediate reasoning without environment interaction.
  • Reflexion represents self-feedback and iterative improvement.
  • Toolformer represents learned tool-use behavior.
  • AutoGPT-style systems illustrate autonomous goal decomposition and tool invocation.
  • Multi-agent frameworks distribute tasks and reasoning across specialized agents.
  • Retrieval-augmented agents externalize knowledge rather than relying solely on model parameters.

10

Results

  • The survey concludes that adding memory, planning, and tools can substantially expand LLM capabilities.
  • Agents enable models to perform tasks requiring multiple environment interactions.
  • Tool use provides access to computation, search, APIs, and domain-specific capabilities.
  • Memory supports persistence and personalization.
  • Planning enables decomposition of complex objectives.
  • Multi-agent collaboration can divide complex work into specialized roles.
  • However, stronger components do not automatically produce reliable end-to-end behavior.
  • Coordination among components becomes a central systems challenge.

11

Conclusion

  • The survey argues that LLM agents are an emerging paradigm for autonomous problem solving.
  • A common architecture combines an LLM controller with profiling, memory, planning, and tools.
  • Interaction with an environment creates a feedback loop unavailable to standalone language models.
  • Agent applications span digital and embodied environments.
  • Major open problems include planning, memory, tool use, evaluation, safety, and reliability.
  • Standardized benchmarks and more rigorous evaluation are needed.
  • The field is expected to move toward more capable, persistent, collaborative, and autonomous agents.
  • The survey provides a taxonomy for reasoning about this rapidly developing research area.

12

References

  • Zhiheng Xi et al., “The Rise and Potential of Large Language Model Based Agents: A Survey,” arXiv:2309.07864.
  • The work is a broad survey of LLM-based agent architectures, applications, and challenges.
  • Primary source: https://arxiv.org/abs/2309.07864.
  • The survey organizes agents around profiling, memory, planning, and action.
  • Representative methods discussed include ReAct, Toolformer, Reflexion, AutoGPT-style agents, and multi-agent systems.
  • Major application areas include software engineering, games, web interaction, robotics, scientific tasks, and social simulation.
  • Key open problems include hallucination, planning, memory, tool use, evaluation, safety, and cost.

Continue reading

Related research

Browse all research