worldweaver | Vision and Autonomy Intelligence Lab

^*Equal contribution.

TL;DR

WorldWeaver (W²) is a streaming multi-agent video diffusion model that explicitly models persistent world states with world state registers (WSR): learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. These registers are grounded with supervision spanning individual agent statistics, a global bird's-eye view, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world-state modeling and visual frame modeling.

Gallery

The rollout examples show synchronized two-player Minecraft generation. WorldWeaver maintains a shared world-state representation so future chunks can condition on persistent state rather than relying only on a limited window of recent frames.

Method Overview

Standard streaming autoregressive diffusion models denoise each new frame from a local frame KV cache. This makes rollout practical, but the model must repeatedly re-infer world information from recent frames, and the stored context remains entangled with visual tokens.

WorldWeaver augments this pipeline with world state registers (WSR): persistent register tokens that carry global scene information and individual agent status across rollout steps. After each generated chunk, the model commits an updated register, removes stale state, and uses the latest register to condition the next frame generation.

At each rollout step, the model updates the register from the previous register, the local context window, and the current action, then uses the committed register to generate the next frame:

\[ \mathbf{r}_i=G_\theta(\mathbf{r}_{i-1},\mathbf{x}_{i-W+1},\ldots,\mathbf{x}_i,a_i),\qquad p_\theta(\mathbf{x}_{i+1}\mid \mathbf{x}_{i-W+1},\ldots,\mathbf{x}_i,a_{i+1},\mathbf{r}_i). \]

During training, WorldWeaver interleaves frame/context tokens and register groups as [C1, R1, C2, R2, ...]. The causal mask makes the rollout causal at the state level: frame tokens attend to the local window and the latest committed register, while each register query attends to the local context ending at its commit step and the immediately preceding register.

To keep world-state modeling from competing with pixel generation, WorldWeaver adopts a Mixture-of-Transformers (MoT) backbone: register tokens and frame tokens are routed through role-specific weight branches, while joint self-attention over the interleaved sequence is preserved so the two pathways keep exchanging information at every step. This separation matters most once the registers carry richer supervised semantics, where a dense backbone that shares parameters across both roles begins to degrade.

WorldWeaver is trained with a three-stage curriculum: (1) Bidirectional training adapts a single-player video prior into a synchronized multi-agent teacher; (2) Causal training converts the teacher into a causal student with the WSR pathway and register supervision; and (3) Self-forcing rolls out the student on its own generated frames and committed registers, exposing state drift together with frame drift to close the train-test gap over long horizons.

Grounding the World State

A core question is not only how to store state, but what the state should represent. WorldWeaver grounds each committed register with auxiliary decoders that make the hidden world state inspectable: per-agent simulator statistics, a global bird’s-eye view, and scene text. These heads are used during training and discarded at inference, so the supervision does not increase rollout cost.

The supervision signals encourage the register to preserve complementary aspects of the world:

Agent states: position, velocity, and orientation give the register explicit per-agent motion targets.
BEV layout: bird’s-eye-view supervision exposes allocentric geometry shared by both players.
Scene text: language targets ask registers to retain categories, attributes, and semantic state.

Experiments

We ablate the supervision signals used to ground the world state registers. The aggregate world score combines visual quality and logical correctness; higher is better.

Variant	Movement		Grounding		Memory		Building		Consistency		World Score ↑
Variant	VLM ↑	FID ↓	VLM ↑	FID ↓	VLM ↑	FID ↓	VLM ↑	FID ↓	VLM ↑	FID ↓	World Score ↑
Baseline	79.7	43.3	81.3	37.2	43.8	61.2	9.4	83.4	57.8	110.7	81.0
Registers only	90.6	43.2	81.3	44.3	62.5	66.1	21.9	84.3	62.5	101.9	93.8
+ Agent stats	95.3	41.0	59.4	45.5	56.3	60.1	9.4	80.8	75.0	107.9	88.1
+ BEV	82.8	39.1	96.9	40.7	46.9	64.7	31.3	74.2	71.9	103.4	102.4
+ Scene text	85.9	40.2	84.4	38.4	62.5	62.1	25.0	78.8	73.4	101.6	103.2
+ All	82.8	34.0	93.8	36.8	46.9	64.8	28.1	75.9	76.6	100.7	105.1

Even without explicit targets, adding registers already lifts the world score from 81.0 to 93.8, because they give the model a dedicated slot to carry cross-agent information instead of recomputing it from the local window at every step. Explicit supervision then determines what that state encodes: the bird’s-eye view is the strongest single signal since it grounds global 3D layout, and combining all three signals on the Mixture-of-Transformers backbone yields the full W² model at 105.1. Crucially, the largest gains fall on state-sensitive categories such as grounding, building, and consistency, showing that persistent registers improve logical coherence across players and rollout steps rather than visual fidelity alone.

Reference

@misc{mo2026worldweaver,
  title={Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers},
  author={Mo, Sicheng and Li, Yuheng and Leng, Ziyang and Singh, Krishna Kumar and Zhou, Bolei},
  year={2026}
}

Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

Sicheng Mo* ¹ , Yuheng Li* ² , Ziyang Leng ¹ , Krishna Kumar Singh ² , Bolei Zhou ¹