Reflexion: Language Agents with Verbal Reinforcement Learning
Key takeaway
Foundational work on self-reflection and episodic memory.
Updated on Oct 3, 2026
01
Abstract
- Reflexion introduces “verbal reinforcement learning”: agents improve through natural-language feedback instead of updating model weights.
- After a trial, the agent generates a textual reflection from the task outcome or evaluator feedback.
- The reflection is stored in episodic memory and supplied to the agent on subsequent trials.
- The framework is evaluated across sequential decision making, coding, and language reasoning.
- On HumanEval, Reflexion reports 91% pass@1, compared with 80% for GPT-4 in the cited comparison.
02
Introduction
- Traditional reinforcement learning updates parameters and can require many environment interactions and expensive optimization.
- Reflexion instead converts reward or feedback into natural-language information that a language model can condition on later.
- This creates a separation between the base model's weights and its task-specific experience.
- Episodic memory acts as a lightweight learning mechanism across trials.
- The approach is motivated by the observation that language models can already interpret, summarize, and reason over textual feedback.
03
Problem
- The central problem is enabling an LLM agent to learn from failed attempts without expensive parameter updates.
- Scalar rewards alone tell an agent whether an outcome was good but often do not explain why it failed.
- Reflexion converts feedback into a richer linguistic explanation that can identify mistakes and suggest future behavior.
- The memory must retain useful information without requiring the agent to replay the entire environment trajectory.
- The framework therefore treats language feedback as a form of policy improvement at inference time.
04
Methodology
- Reflexion supports external feedback such as environment rewards and internal feedback generated by the language model.
- Reflection can be generated after every trial or only when the agent fails, depending on the task.
- Memory is not a parameter update; it is textual context that changes the next inference.
- The method can therefore reuse the same frozen language model across repeated attempts.
- Different tasks use different evaluators because success criteria vary between games, code execution, and language reasoning.
05
Architecture
- The architecture separates acting, evaluating, reflecting, and remembering into distinct functional stages.
- The actor sees the task and relevant memory before generating the next attempt.
- The evaluator supplies an objective or semi-objective signal after execution.
- The reflection stage summarizes the trajectory and feedback into actionable natural language.
- Episodic memory stores reflections so later attempts can avoid previously identified mistakes.
- The resulting loop is: Act → Evaluate → Reflect → Remember → Act again.
06
Dataset
- The paper uses several task families to test whether verbal feedback improves subsequent behavior.
- In sequential decision-making, the agent operates in environments where repeated trials expose it to success or failure feedback.
- In coding, HumanEval provides executable unit tests that act as an external evaluator.
- In language reasoning, feedback can be generated by comparing answers against known solutions or using an evaluator model.
- The benchmark selection intentionally covers both objectively verifiable and more open-ended tasks.
07
Training
- Reflexion does not require gradient-based training for the agent's iterative improvement.
- The model instead receives demonstrations and task instructions through prompts and accumulates reflections in context.
- Different experiments use different base models and prompting configurations.
- Coding experiments use execution feedback from unit tests to ground reflection.
- The approach therefore shifts much of the learning loop from parameter optimization to inference-time context construction.
08
Experiments
- The experiments compare agents with and without reflective memory.
- Coding experiments test whether feedback from failed unit tests can be transformed into useful future corrections.
- Sequential decision-making experiments examine repeated attempts under a fixed trial budget.
- Language reasoning experiments compare different feedback sources and reflection strategies.
- The paper additionally studies how the amount and type of feedback influence performance.
09
Baselines
- Baselines include standard language-model agents without episodic reflection.
- ReAct-style agents provide a comparison for agents that reason and act but do not explicitly convert trial outcomes into persistent verbal lessons.
- Coding comparisons include strong contemporary language-model baselines such as GPT-4.
- Ablations compare scalar feedback, verbal feedback, and different memory/reflection configurations.
- The goal is to distinguish improvement caused by simply retrying from improvement caused by useful accumulated experience.
10
Results
- On HumanEval, Reflexion achieves 91% pass@1 in the paper's reported setup, exceeding the cited GPT-4 result of 80%.
- The coding agent benefits from converting compiler and unit-test failures into explicit textual guidance for the next attempt.
- Sequential decision-making results show that reflection can substantially improve success over memory-free agents across repeated trials.
- Language-reasoning experiments demonstrate that verbal feedback can act as a reusable intermediate representation of previous mistakes.
- The gains occur without changing the underlying model parameters between trials.
11
Conclusion
- Reflexion demonstrates a lightweight alternative to parameter-level reinforcement learning for improving language agents.
- Its core contribution is turning feedback into natural-language episodic memory that the same model can use on future trials.
- The method is model-agnostic with respect to the exact feedback source: rewards, test results, or language critiques can all drive reflection.
- The HumanEval result shows that repeated inference-time learning can produce large gains on executable coding tasks.
- The work helped establish verbal feedback and reflective memory as important building blocks for modern agent architectures.
12
References
Primary source: Shinn, Noah; Cassano, Federico; Berman, Edward; Gopinath, Ashwin; Narasimhan, Karthik; Yao, Shunyu. “Reflexion: Language Agents with Verbal Reinforcement Learning.” arXiv:2303.11366.
- The paper was published at NeurIPS 2023.
- The central distinction is between traditional reinforcement learning, which changes parameters, and Reflexion, which changes the agent's textual context.
- The paper evaluates sequential decision-making, coding, and language reasoning rather than a single benchmark.
- The authors include ablation and analysis studies covering feedback signals, feedback incorporation, and agent types.
Source trail
Primary papers, implementation pages, and external source records used to ground this explainer.
Continue reading
Related research
GitSwarm
GitSwarm turns inference-time work into a persistent, branchable body of reusable contributions that later agents can build on.
AI AgentsLongHarness
LongHarness helps agents reason over 100M-token document collections by keeping evidence and conclusions as explicit, revisable states that guide one another.
AI AgentsGenerative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.