World In-Context Learning (WICL): RoboScript Teaches Robots from One Demo

World In-Context Learning (WICL) is a paradigm for agentic robot manipulation in which a frozen foundation model learns a task from a single demonstration by rehearsing in a simulator and writing feedback into a reusable context memory rather than updating model weights. The authors implement WICL in RoboScript, which builds a Task Memory from object roles, task stages, relative grasp/place specifications, and lessons from rehearsal. They report 83.4% mean success across all 50 RoboTwin 2.0 tasks with one demonstration per task and no gradient updates, and demonstrate transfer to a physical dual-arm robot through cross-embodiment memory adaptation.

A robot can imitate a demonstration and still fail in a new scene. A video may show an object being grasped at a particular pixel location, but that location is not necessarily correct when the object moves, the camera changes, or the table height is different.

A trained action policy can learn robust mappings from observations to actions, but the skill is stored in model parameters and usually requires demonstrations and training updates. In-context learning avoids updating weights, but a demonstration recorded in another scene does not reveal how the robot's actions will affect the current world.

WICL adds the missing feedback loop. The robot rehearses in an interactive world, observes the consequences of actions, and updates a task-specific memory. Instead of memorising a single trajectory, it learns a semantic procedure grounded in repeated interaction.

WICL connects several active research directions.

Vision-language-action models (VLAs) map visual and language inputs to robot actions. WICL instead emphasises simulator feedback and reusable context memory for task-specific adaptation.

World action models (WAMs) predict or represent future world states to support action selection. WICL uses an interactive world as a rehearsal environment, then stores learned task knowledge in context.

In-context learning adapts model behaviour through examples and prompt context without changing model parameters. WICL extends this idea with feedback from simulated interaction.

Sim-to-real transfer seeks to make skills learned in simulation work on physical hardware. WICL's semantic task memory and data-isolation rules are intended to support this transfer.

The broader research question is whether task competence can be stored as a portable, inspectable memory that a frozen foundation model can execute across different scenes and embodiments.

For related approaches to persistent AI agent memory, see WikiTrace: Evidence-Linked Memory for Reliable AI Agents. For targeted language-model weight repair, see Tree-Ising: Gradient-Free LLM Weight Editing Explained. These papers address different adaptation layers: stored experience, embodied task context, and model parameters.

The paper's core observation is that task knowledge can be represented in terms of object roles and action relationships rather than absolute visual coordinates.

For example, a task memory might say: identify the handle, approach from its left side, grasp near its midpoint, lift clear of the fixture, and place the object relative to the target surface. It should not simply memorise a pixel coordinate or a world-space pose from the demonstration scene.

This semantic representation is intended to survive changes in appearance and layout. The robot still needs perception and motion planning to ground the roles and relative specifications in the current scene, but the task procedure can be reused.

The distinction is important: WICL does not claim that the robot can ignore vision or geometry. It separates reusable task structure from scene-specific grounding, then lets the robot re-ground the task in each new environment.

RoboScript is the infrastructure the authors build to implement WICL for agentic manipulation. Its main components are a frozen foundation model, an initial demonstration, a simulator for rehearsal, a Task Memory representation, perception and motion-planning interfaces, and a deployment process that uses the frozen memory on new scenes.

The system converts the demonstration into an initial task procedure. It then rehearses that procedure in simulation, receives action outcomes and observations, diagnoses failures, and edits the task memory. The resulting memory is frozen before evaluation on held-out deployment scenes.

The architecture deliberately separates reusable skill knowledge from environment-specific execution. A task memory can describe object roles, stage order, and relative grasp or placement requirements. Robot-specific bindings supply details such as joint limits, table height, and gripper configuration.

The learning process can be understood as a sequence of stages:

• Demonstration: observe one example of the manipulation task.
• Initial memory: convert the demonstration into a semantic task procedure.
• Rehearsal: execute the procedure in simulator scenes and collect observations and action outcomes.
• Diagnosis: identify why a stage succeeded or failed using feedback from the environment.
• Memory revision: update the task procedure, relative specifications, and lessons.
• Validation: test the revised memory on additional rehearsal scenes.
• Freeze: stop changing the memory before deployment evaluation.
• Deployment: use the frozen memory in unseen scenes, with only the observations available to the real robot.

The freeze boundary matters. It prevents the system from using test-time privileged state to revise the memory after seeing the evaluation scene. The paper emphasises data isolation between rehearsal and deployment.

A practical WICL-inspired architecture can be built from a frozen vision-language model, a simulator, a task-memory compiler, a perception and tracking stack, a robot action API, and an evaluation harness.

Start by choosing a task with a clear success predicate and a simulator that supports repeatable resets. Record one high-quality demonstration and convert it into a semantic plan: object roles, stages, preconditions, relative grasps, relative placements, and completion conditions.

Run rehearsal episodes across varied scenes. Capture actual action receipts, state observations, planner errors, and task outcomes. Use failures to update a candidate memory, but accept an edit only after it passes a held-out rehearsal scene. Freeze the final memory before deployment evaluation.

Keep all simulator-only fields out of the deployed memory. Log memory versions, robot calibration, scene seeds, perception confidence, planner outcomes, and recovery attempts. Test safety constraints separately from task success.

WICL is a new preprint and its results require independent reproduction. The headline benchmark is RoboTwin 2.0, and the physical transfer demonstration does not establish reliability across a broad range of robots, objects, and real-world settings.

The approach depends on simulator quality. If contact dynamics, friction, compliance, sensing, or motion-planning outcomes differ substantially from reality, rehearsal feedback may produce a memory that fails at deployment.

The task-memory linter can reject explicit coordinates and instance identifiers, but semantic text can still encode hidden assumptions. A rule such as “grasp the handle from the left” may not transfer when the handle geometry or robot approach direction changes.

The approach also depends on reliable perception, tracking, action receipts, and robot-specific bindings. A correct high-level plan cannot compensate for a poor grasp controller or inaccurate object grounding.

Open questions include scaling to long-horizon tasks, continuous manipulation, deformable objects, multi-robot collaboration, safety-critical workcells, and online changes in the physical environment.

World In-Context Learning proposes a compelling alternative to retraining a robot policy for every new task. A frozen model starts from one demonstration, rehearses in simulation, and stores feedback as a reusable Task Memory.

RoboScript's most important design ideas are semantic task memory, simulation rehearsal, action receipts, data isolation, and freezing the learned memory before deployment. The authors report 83.4% mean success across all 50 RoboTwin 2.0 tasks with one demonstration per task and no gradient updates, plus a physical dual-arm transfer demonstration.

For engineers, the strongest takeaway is that a simulator can serve as an interactive teacher, not just a place to test a finished policy. The main caveats are simulator fidelity, perception and control dependencies, benchmark scope, and the need for broader physical replication.

Primary keywords: World In-Context Learning, WICL, RoboScript, agentic robot manipulation, robot learning from one demonstration, robot task memory.

Technical keywords: simulation-based robot learning, embodied AI, robot foundation models, vision-language-action models, world action models, frozen foundation model, in-context learning for robotics, sim-to-real transfer, cross-embodiment robot learning.

Long-tail keywords: how robots learn manipulation from one demonstration; teach a robot without fine-tuning; simulator rehearsal for robot learning; task memory for embodied agents; how to transfer robot skills across scenes; AI agents for robotic manipulation; RoboTwin 2.0 benchmark explained.