Research

Reflexion: Language Agents with Verbal Reinforcement Learning

Noah Shinn12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on self-reflection and episodic memory.

Updated on Oct 3, 2026

01

Abstract

  • Reflexion introduces “verbal reinforcement learning”: agents improve through natural-language feedback instead of updating model weights.
  • After a trial, the agent generates a textual reflection from the task outcome or evaluator feedback.
  • The reflection is stored in episodic memory and supplied to the agent on subsequent trials.
  • The framework is evaluated across sequential decision making, coding, and language reasoning.
  • On HumanEval, Reflexion reports 91% pass@1, compared with 80% for GPT-4 in the cited comparison.

02

Introduction

  • Traditional reinforcement learning updates parameters and can require many environment interactions and expensive optimization.
  • Reflexion instead converts reward or feedback into natural-language information that a language model can condition on later.
  • This creates a separation between the base model's weights and its task-specific experience.
  • Episodic memory acts as a lightweight learning mechanism across trials.
  • The approach is motivated by the observation that language models can already interpret, summarize, and reason over textual feedback.

03

Problem

  • The central problem is enabling an LLM agent to learn from failed attempts without expensive parameter updates.
  • Scalar rewards alone tell an agent whether an outcome was good but often do not explain why it failed.
  • Reflexion converts feedback into a richer linguistic explanation that can identify mistakes and suggest future behavior.
  • The memory must retain useful information without requiring the agent to replay the entire environment trajectory.
  • The framework therefore treats language feedback as a form of policy improvement at inference time.

04

Methodology

  • Reflexion supports external feedback such as environment rewards and internal feedback generated by the language model.
  • Reflection can be generated after every trial or only when the agent fails, depending on the task.
  • Memory is not a parameter update; it is textual context that changes the next inference.
  • The method can therefore reuse the same frozen language model across repeated attempts.
  • Different tasks use different evaluators because success criteria vary between games, code execution, and language reasoning.
Figure 1

05

Architecture

  • The architecture separates acting, evaluating, reflecting, and remembering into distinct functional stages.
  • The actor sees the task and relevant memory before generating the next attempt.
  • The evaluator supplies an objective or semi-objective signal after execution.
  • The reflection stage summarizes the trajectory and feedback into actionable natural language.
  • Episodic memory stores reflections so later attempts can avoid previously identified mistakes.
  • The resulting loop is: Act → Evaluate → Reflect → Remember → Act again.

06

Dataset

  • The paper uses several task families to test whether verbal feedback improves subsequent behavior.
  • In sequential decision-making, the agent operates in environments where repeated trials expose it to success or failure feedback.
  • In coding, HumanEval provides executable unit tests that act as an external evaluator.
  • In language reasoning, feedback can be generated by comparing answers against known solutions or using an evaluator model.
  • The benchmark selection intentionally covers both objectively verifiable and more open-ended tasks.

07

Training

  • Reflexion does not require gradient-based training for the agent's iterative improvement.
  • The model instead receives demonstrations and task instructions through prompts and accumulates reflections in context.
  • Different experiments use different base models and prompting configurations.
  • Coding experiments use execution feedback from unit tests to ground reflection.
  • The approach therefore shifts much of the learning loop from parameter optimization to inference-time context construction.

08

Experiments

  • The experiments compare agents with and without reflective memory.
  • Coding experiments test whether feedback from failed unit tests can be transformed into useful future corrections.
  • Sequential decision-making experiments examine repeated attempts under a fixed trial budget.
  • Language reasoning experiments compare different feedback sources and reflection strategies.
  • The paper additionally studies how the amount and type of feedback influence performance.

09

Baselines

  • Baselines include standard language-model agents without episodic reflection.
  • ReAct-style agents provide a comparison for agents that reason and act but do not explicitly convert trial outcomes into persistent verbal lessons.
  • Coding comparisons include strong contemporary language-model baselines such as GPT-4.
  • Ablations compare scalar feedback, verbal feedback, and different memory/reflection configurations.
  • The goal is to distinguish improvement caused by simply retrying from improvement caused by useful accumulated experience.

10

Results

  • On HumanEval, Reflexion achieves 91% pass@1 in the paper's reported setup, exceeding the cited GPT-4 result of 80%.
  • The coding agent benefits from converting compiler and unit-test failures into explicit textual guidance for the next attempt.
  • Sequential decision-making results show that reflection can substantially improve success over memory-free agents across repeated trials.
  • Language-reasoning experiments demonstrate that verbal feedback can act as a reusable intermediate representation of previous mistakes.
  • The gains occur without changing the underlying model parameters between trials.

11

Conclusion

  • Reflexion demonstrates a lightweight alternative to parameter-level reinforcement learning for improving language agents.
  • Its core contribution is turning feedback into natural-language episodic memory that the same model can use on future trials.
  • The method is model-agnostic with respect to the exact feedback source: rewards, test results, or language critiques can all drive reflection.
  • The HumanEval result shows that repeated inference-time learning can produce large gains on executable coding tasks.
  • The work helped establish verbal feedback and reflective memory as important building blocks for modern agent architectures.

12

References

Primary source: Shinn, Noah; Cassano, Federico; Berman, Edward; Gopinath, Ashwin; Narasimhan, Karthik; Yao, Shunyu. “Reflexion: Language Agents with Verbal Reinforcement Learning.” arXiv:2303.11366.

  • The paper was published at NeurIPS 2023.
  • The central distinction is between traditional reinforcement learning, which changes parameters, and Reflexion, which changes the agent's textual context.
  • The paper evaluates sequential decision-making, coding, and language reasoning rather than a single benchmark.
  • The authors include ablation and analysis studies covering feedback signals, feedback incorporation, and agent types.

Continue reading

Related research

Browse all research