ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
ReasoningBank turns agent successes and failures into reusable reasoning strategies that improve future tasks without updating model weights.
Key takeaway
The system retrieves relevant memory, judges outcomes, extracts strategy or failure lessons, and consolidates them into an evolving memory bank; MaTTS makes additional test-time exploration feed higher-quality memories back into the loop.
Updated on Oct 2, 2026
01
Abstract
ReasoningBank turns agent experience into reusable reasoning memory. Instead of retaining raw trajectories, it distills successful strategies and failure lessons, retrieves relevant memories before future tasks, and consolidates new experience into an evolving memory bank.
02
Introduction
Persistent agents repeatedly encounter related tasks but normally do not convert experience into reusable reasoning knowledge. ReasoningBank adds a learning layer between past trajectories and future decisions.
03
Problem
Raw trajectories are long and noisy, while success-only memory discards lessons contained in failures. The system needs compact, transferable reasoning knowledge that can be retrieved when a future task resembles a previous one.
04
Background
The approach connects test-time learning, memory-augmented agents, embedding retrieval, LLM-based trajectory evaluation, and self-improvement. MaTTS extends this idea by making additional test-time trajectories contribute to memory quality.
05
Methodology
For each task, relevant memory items are retrieved with embedding similarity and injected into the agent context. After execution, an LLM judge labels the trajectory, useful strategies and pitfalls are extracted, and the memory bank is consolidated.
ReasoningBank cumulative improvement
Original arXiv figure showing ReasoningBank cumulative successful tasks compared with no memory.
arXiv HTML v2
ReasoningBank closed loop
Original arXiv figure showing memory retrieval, extraction, LLM judging, and consolidation.
arXiv HTML v2
MaTTS
Original arXiv figure comparing vanilla test-time scaling with parallel and sequential memory-aware scaling.
arXiv HTML v2
Formula notes
Mathematical details
Agent policy with memory
The agent policy is parameterized by the backbone LLM and conditioned on the memory module and available actions.
- A
- -action space
- L
- -backbone language model
- M
- -ReasoningBank memory
Environment transition
The environment transition maps the current state and chosen action to the next state.
- T
- -environment transition function
- a_t
- -action selected at time t
- s_t
- -state at time t
MaTTS scaling factor
k controls how much additional test-time experience is generated for memory-aware scaling.
- k
- -parallel trajectory count or sequential refinement count
06
Architecture
ReasoningBank has three core stages: memory retrieval, memory extraction, and memory consolidation. Stored items contain a title, description, and reasoning content. MaTTS adds parallel self-contrast or sequential self-refinement.
07
Dataset
Experiments cover WebArena, Mind2Web, and SWE-Bench-Verified using Gemini-2.5 and Claude-3.7. WebArena includes Shopping, Admin, GitLab, Reddit, and Multi subsets.
08
Training
The system is primarily an inference-time learning mechanism rather than foundation-model retraining. Experience is converted into reusable memory during task execution and later retrieved for related tasks.
09
Experiments
The reported experiments evaluate memory-augmented agents across WebArena, Mind2Web, and SWE-Bench-Verified, with gains reported across the evaluated backbone models and datasets.
10
Baselines
Baselines include the underlying agents without ReasoningBank memory and alternative test-time scaling approaches.
11
Results
The central result is that abstracted strategies and failure lessons transfer across tasks, while MaTTS makes additional test-time computation more useful by converting it into durable memory.
12
Ablation
Memory retrieval, judging, extraction, consolidation, and MaTTS scaling are the key components whose removal changes memory quality or reuse.
13
Limitations
Memory quality depends on judging and extraction, retrieval can surface irrelevant memories, and contradictory or stale memories can accumulate.
14
Conclusion
ReasoningBank turns agent experience into a reusable reasoning asset and provides continual test-time improvement without changing model weights.
15
References
ReasoningBank primary paper and related work on memory-augmented agents, test-time scaling, self-improvement, WebArena, Mind2Web, SWE-Bench-Verified, and MaTTS.
Source trail
Primary papers, implementation pages, and external source records used to ground this explainer.
Synapse
Trajectory-oriented agent memory baseline.
Agent Workflow Memory
Workflow-based success-memory baseline discussed by the paper.
ReAct: Synergizing Reasoning and Acting in Language Models
Agent execution style used in the experimental systems.
01arXiv v202arXiv HTML v203Official GitHub repositoryContinue reading
Related research
GitSwarm
GitSwarm turns inference-time work into a persistent, branchable body of reusable contributions that later agents can build on.
AI AgentsLongHarness
LongHarness helps agents reason over 100M-token document collections by keeping evidence and conclusions as explicit, revisable states that guide one another.
AI AgentsGenerative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.