MachinoAI Research
Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Use past discovery trees as a replay environment for improving exploration policies cheaply.
TL;DR
Dream-RSI alternates online exploration, replay construction and offline policy improvement; it reports lower discovery cost across several domains.
Why It Matters
The production insight is to turn expensive agent trajectories into reusable evaluation infrastructure.
Research Brief
The shortest useful explanation.
Dream-RSI alternates online exploration, replay construction and offline policy improvement; it reports lower discovery cost across several domains.
Core Explanation
Dream-RSI treats accumulated discovery histories as replay simulators so exploration policies can be improved offline before online deployment.
Why It Matters
The production insight is to turn expensive agent trajectories into reusable evaluation infrastructure.
Dream-RSI overview
Original paper figure.
Section 01
The Problem
01 THE PROBLEM
Long-horizon discovery needs many proposal/evaluation cycles. Fixed exploration policies cannot adapt; online meta-policy optimization is expensive because candidates require long rollouts before feedback.
Section 02
The Big Idea
02 THE BIG IDEA
Completed discovery history becomes a replay simulator. Alternative policies can traverse recorded branches and reuse recorded execution outcomes.
Section 03
How It Works
03 HOW IT WORKS
Online exploration creates a discovery tree; simulator construction creates a replay pool; dreaming proposes and evaluates alternative policies offline; the selected policy is redeployed online and creates new history.
Section 04
The Math
04 THE MATH
The paper defines eligible nodes, parallel action batches and a Pareto reward balancing attainment against parallelism cost.
Section 05
Architecture
05 ARCHITECTURE
A fixed discovery agent is controlled by an exploration-policy layer. Attempts form a tree; nodes retain state, artifacts, diagnostics and scores; a replay pool supports offline policy evaluation.
Section 06
Experiments
06 EXPERIMENTS
Eight scientific discovery tasks cover algorithm engineering, mathematical optimization and GPU kernel engineering. Lasso results: Gemini-3.1-Pro fixed uses 550 calls and 3587.1 ms average downstream runtime; Dream-RSI uses 317 calls and 2931.0 ms. Gemini-3.7-Flash fixed uses 3200 calls and 2516.7 ms; Dream-RSI uses 1879 calls and 2350.6 ms. Reported: up to 162x fewer calls than SimpleTES, over 50x budget savings on several optimization tasks, and 1.79x–2.43x fewer generations or up to 2.09x kernel-performance improvement depending on comparison.
Section 07
What We Learned
07 WHAT WE LEARNED
Replay makes controller evaluation cheap, but the simulator is limited to realized history. Production systems should version discovery trees and measure replay coverage before trusting offline improvements. Offline gains require online validation.
Section 08
What Came Next
08 WHAT CAME NEXT
Related work includes model-based RL/world models, recursive self-improvement, agentic discovery and exploration-policy optimization, including SimpleTES. Future work should strengthen replay coverage and offline-to-online consistency.
Section 09
Engineering Takeaways
ENGINEERING TAKEAWAYS
Store trajectories as structured state with decisions, tools, artifacts, evaluator outputs and costs. Build replay evaluators for controller changes but keep online evaluation authoritative. Watch stale histories, coverage gaps, controller overfitting and incorrect cost models.
Formulas
Parallel exploration action set
An exploration action is a batch of eligible nodes expanded in parallel.
Pareto reward
Replay policy quality balances attainment and efficient probing.
Related Research
Generative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.
AI AgentsMIRA
MIRA is an autonomous medical agent evaluated in a sandboxed EHR workflow with tools for diagnosis, testing, treatment, and admission decisions.
AI AgentsAgentic Reasoning
A tool-using agent framework that extends LLM reasoning with web search, coding, and structured reasoning memory for deep research tasks.