Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Dream-RSI turns completed discovery trajectories into a replayable simulator. Alternative exploration policies can be evaluated offline against recorded histories before selected policies are redeployed for new online discovery.

Long-horizon scientific discovery requires repeated proposal and evaluation cycles. The central challenge is improving exploration without paying the full cost of repeatedly executing expensive discovery trajectories.

Fixed exploration policies cannot adapt efficiently, while online meta-policy optimization is expensive because candidate policies require long rollouts before meaningful feedback becomes available.

The approach connects agentic scientific discovery with model-based reinforcement learning, replay-based evaluation, recursive self-improvement, and exploration-policy optimization.

Online exploration creates a discovery tree. Completed branches become replayable history. Alternative policies are evaluated offline, a promising policy is selected, and the updated policy is deployed online.

The architecture contains a fixed discovery agent, an exploration-policy layer, a discovery tree containing states and artifacts, and a replay pool used for offline controller evaluation.

Experiments cover eight scientific discovery tasks spanning algorithm engineering, mathematical optimization, and GPU kernel engineering.

The improvement target is the exploration policy rather than the underlying foundation model. Offline replay is used to evaluate policy changes cheaply before online deployment.

The principal comparisons are fixed exploration and SimpleTES, with reported comparisons including up to 162x fewer calls than SimpleTES and more than 50x budget savings on several optimization tasks.

Reported experiments show substantial reductions in exploration calls and budget in several tasks, with gains ranging from fewer generations to improved GPU-kernel performance depending on the comparison.

The key factors are replay coverage, policy selection, online validation, and the quality of recorded trajectories; the stored material does not provide a complete component-by-component ablation.

Replay can only evaluate behavior represented in historical trajectories. Stale histories, coverage gaps, controller overfitting, and offline-to-online disagreement can produce misleading improvements.

Dream-RSI provides a practical route to recursive improvement of exploration policies by converting expensive discovery history into reusable offline evaluation data. Online validation remains authoritative.

Dream-RSI primary paper and cited work on scientific discovery agents, model-based reinforcement learning, recursive self-improvement, exploration-policy optimization, and SimpleTES.