MachinoAI Research
ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
ReasoningBank turns agent successes and failures into reusable reasoning strategies that improve future tasks without updating model weights.
TL;DR
The system retrieves relevant memory, judges outcomes, extracts strategy or failure lessons, and consolidates them into an evolving memory bank; MaTTS makes additional test-time exploration feed higher-quality memories back into the loop.
Why It Matters
Persistent agents need a learning substrate between model weights and raw logs. ReasoningBank shows how structured memory can convert experience—including failures—into reusable operational knowledge and combine with test-time scaling.
Research Brief
The shortest useful explanation.
The system retrieves relevant memory, judges outcomes, extracts strategy or failure lessons, and consolidates them into an evolving memory bank; MaTTS makes additional test-time exploration feed higher-quality memories back into the loop.
Core Explanation
ReasoningBank distills reusable reasoning strategies from both successful and failed agent trajectories, retrieves relevant memories for future tasks, and consolidates new memory items after each task. MaTTS extends test-time scaling by using multiple or sequential trajectories as contrastive signals for better memory construction.
Why It Matters
Persistent agents need a learning substrate between model weights and raw logs. ReasoningBank shows how structured memory can convert experience—including failures—into reusable operational knowledge and combine with test-time scaling.
ReasoningBank cumulative improvement
Original arXiv figure showing ReasoningBank cumulative successful tasks compared with no memory.
ReasoningBank closed loop
Original arXiv figure showing memory retrieval, extraction, LLM judging, and consolidation.
MaTTS
Original arXiv figure comparing vanilla test-time scaling with parallel and sequential memory-aware scaling.
Section 01
Problem
01. Problem
Persistent agents repeatedly encounter related tasks but usually do not convert experience into reusable reasoning knowledge. Raw trajectories are long and noisy, while success-only workflow memory discards lessons contained in failures. ReasoningBank addresses the missing learning layer by distilling strategies and failure lessons into structured memory items.
Section 02
How It Works
03. How It Works
For each streaming task, the agent retrieves top-k relevant memory items using embedding similarity and injects them into the system instruction. After completion, an LLM-as-a-judge labels the trajectory as success or failure. Successful trajectories yield validated strategies; failures yield pitfalls and counterfactual lessons. The resulting items are consolidated into the memory bank. MaTTS performs parallel self-contrast or sequential self-refinement so extra trajectories create stronger memory.
Section 03
Architecture
05. Architecture
ReasoningBank has three operational stages around each task: memory retrieval, memory extraction, and memory consolidation. The stored item has title, description, and content. The implementation intentionally keeps retrieval as embedding-based top-k similarity and consolidation simple to isolate memory-content quality. MaTTS adds either parallel self-contrast or sequential self-refinement.
Section 04
Experiments
06. Experiments
The authors evaluate on WebArena, Mind2Web, and SWE-Bench-Verified using Gemini-2.5 and Claude-3.7. WebArena covers Shopping, Admin, GitLab, Reddit, and Multi subsets. Metrics include success rate and average steps. The paper reports consistent gains across backbone models and datasets.
Formulas
Agent policy with memory
The agent policy is parameterized by the backbone LLM and conditioned on the memory module and available actions.
Environment transition
The environment transition maps the current state and chosen action to the next state.
MaTTS scaling factor
k controls how much additional test-time experience is generated for memory-aware scaling.
Related Research
Generative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.
AI AgentsMIRA
MIRA is an autonomous medical agent evaluated in a sandboxed EHR workflow with tools for diagnosis, testing, treatment, and admission decisions.
AI AgentsAgentic Reasoning
A tool-using agent framework that extends LLM reasoning with web search, coding, and structured reasoning memory for deep research tasks.