InWorld is an interactive world you can see and hear. You say what happens next. A world agent decides how the story moves, and a fast audiovisual generator renders it as video with sound. Each new segment is checked against the plan and written back into the world, so later turns build on what actually appeared on screen.
The agent, Murea-Agent, keeps track of characters, objects, places, and narrative progress. The generator, Murea, comes in two configurations: Murea-Pro, the flagship model for multi-turn interaction, and Murea-Fast, a lightweight few-step model built on LTX-2.5. Branching histories let you return to an earlier decision and try another path, and co-op mode puts several players' actions into the same shot.
Read the abstract
Interactive world models aim to simulate environments that evolve under user actions while retaining the consequences of past interactions. Extending such worlds to sight and sound requires persistent entities, changing scene states, and local audiovisual dynamics to remain coherent across turns. This paper presents InWorld, an interactive audiovisual world model that couples explicit world-state management with neural audiovisual generation. Its world agent, Murea-Agent, maintains characters, objects, locations, and narrative progress, interprets user actions in context, and compiles planned events into generation conditions. Observations of generated frames inform subsequent planning and selected state updates, closing the interaction loop. Its audiovisual generator, Murea, realizes these events through Omni-Reference Generation, using visual references to anchor entity appearance and structured descriptions to specify actions and dialogue. Streaming generation uses modality-specific histories, with longer audio context for unfinished utterances. Degraded-history training prepares the model for imperfect rollout context, while sequential MeanFlow and distribution matching distillation enable few-step synthesis. Branching histories support alternative continuations, and co-op interaction brings multiple participants' actions into the same world.
We evaluate the generation components in complementary configurations. Murea-Pro achieves an Overall score of 81.13 on WBench Full-289, the highest among the compared systems. Murea-Fast achieves a Total score of 64.98 on OpenS2V-Eval and an aggregate score of 69.9 on OS2V-Pro, an extended evaluation of reference fidelity, attribute binding, rendering quality, and audio.
Generated by InWorld
Every world. A story unfolding.
Follow each world from one moment to the next. Characters, places, and actions unfold along a single story path. Read each sequence from left to right, and scroll to discover what happens next.
Continue below for shared adventures, longer journeys, and more visual worlds.
Shape your story in every world
Frames generated by InWorld, from the report's overview figure. The quoted lines are what players typed; each one became the next segment of the story.
Snowbound outpostDark magic academyFlooded Gusu
An ongoing journey
One story on the star-sea island, continued turn by turn.
Turn 1“Climb aboard.”
Turn 2“Keep going.”
Turn 3“@Mechanic, keep flying home.”
A world you can change
Actions leave lasting effects that later segments respect.
Turn 1“Enter the code.”
Turn 2“Unplug it.”
Turn 3“Look again.”The screen is still dark.
Everyday adventures
Small moments in a harbour town.
Turn 1“A warm bun, please.”
Turn 2“And soy milk.”
A shared adventure
In co-op mode, each player's action is bound to their own character and rendered in one shared shot.
P1“Go around.”
P2“I'll hold it off.”
Next roundEscape together
Shared adventures, turn by turn
Follow four co-op stories as characters explore, face danger, and move forward together.
Co-op expedition · 22 connected turns
From dragon vault to canyon bridge 巨龙的宝库
01 Turn 01
洛伊 · 森林守卫将Establish this character at a distinct position (1) in the opening scene, looking around the environ；浅月 · 持剑游侠将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
Scene context
本轮行动：洛伊 · 森林守卫将Establish this character at a distinct position (1) in the opening scene, looking around the environ；浅月 · 持剑游侠将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
洛伊 · 森林守卫将Establish this character at a distinct position (1) in the opening scene, looking around the environ；洛伊 · 旅行法师将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
Scene context
本轮行动：洛伊 · 森林守卫将Establish this character at a distinct position (1) in the opening scene, looking around the environ；洛伊 · 旅行法师将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
Ying· 精灵术士将Establish this character at a distinct position (1) in the opening scene, looking around the environ；Jack· 持剑游侠将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
Scene context
本轮行动：Ying· 精灵术士将Establish this character at a distinct position (1) in the opening scene, looking around the environ；Jack· 持剑游侠将Establish this character at a distinct position (2) in the opening scene, looking around the environ。
The generator keeps no memory between calls, and there is no simulator behind the world. So Murea-Agent keeps an explicit, inspectable world state outside the generator, plans in natural language, and checks every rendered segment against what it intended.
The planner, a multimodal language model, reads a snapshot of the world state, your instruction, a duration budget for one segment, and the final frame of the previous segment. The symbolic state says who exists and what they look like; the frame shows how the last segment actually ended.
It returns a directive: the location, the participating assets, the event, dialogue and speakers, how the new segment connects to its parent, and proposed state changes. A request that needs more than one segment is split into ordered events that fit the budget, and only the first group is rendered now.
“Go to the market and talk to the guard.” This turn shows the walk to the market. The conversation with the guard is rewritten as a complete instruction and becomes the input to the next turn. You can interrupt at any point.
Step 2 of 4
Compile the conditions
A deterministic compiler turns the directive into generator conditions: prompt text, ordered reference images, and voice references. Selected characters go first, then other characters and focal objects, then the location. Voices follow who is speaking, never a reference's position, so a place or an object cannot be handed a voice by accident.
Prompts use the six-section format the generator was trained on: subject definitions, a summary, reference-retention statements, the shot, the soundscape, and music. The compiler also decides whether to continue from the parent segment's latents or cut to a new shot.
The market is new, so it is registered with a generated reference image before it first appears, and its soundscape is fixed on this first visit for reuse later. Walking into it is a continuous move, so the segment continues from the previous one. A jump to a distant known place would start a new shot from that place's reference image.
Step 3 of 4
Render video and sound together
Murea jointly generates the segment's video and audio. Reference images anchor each entity's appearance, optional voice samples supply timbre, and on continuation the parent's tail latents carry pose, motion, camera, and layout across the boundary.
Planning happens before rendering, so its latency adds directly to waiting time. The planner emits the fields needed for rendering first, and directives for the suggested next actions are prepared in the background, so choosing a suggestion skips a planning call.
The parent's latents keep the character's pose and the camera moving smoothly out of the previous segment, while the registered reference image keeps the character recognisable on the way to the market.
Step 4 of 4
Observe what actually happened
Whether a requested change happened can only be seen in the frames. While the segment plays, an observation model compares its final frame with the planned event and the current characters and inventory, then updates next-turn suggestions, the objective, and which items were picked up or dropped.
Observation runs asynchronously, so playback never waits for it. A result is applied only if the world is still at the version that was observed. If you have already continued, undone a turn, or edited the world, the stale result is discarded.
\[\hat{o}_t=\rho\big(F_t,\,D_t,\,\mathcal{E}_t\big),\qquad \mathcal{W}\leftarrow\mathcal{U}_{\mathrm{obs}}\big(\mathcal{W},\,\hat{o}_t\big)\ \text{ only if }\ \mathrm{ver}(\mathcal{W})=\mathrm{ver}(\mathcal{W}_t)\]
In the example
The final frame is checked against the planned walk. Items the character visibly picked up, like the backpack in the overview figure below, are added to the inventory, and the conversation with the guard waits as the next turn.
One turn: instruction \(u_t\), state snapshot \(\Sigma\), previous final frame \(F_{t-1}\), directive \(D_t\), generator condition \(c_t\), optional parent latents \(\tilde{Z}_t\), rendered segment \(X_t\).
What the world remembers
Persistent symbolic state is the long-range memory; audiovisual latents are the short-range context.
\(\mathcal{E}\)
Asset registryEvery character, object, and location, each with a reference image, an optional voice sample, and a locked description that is reused word for word. Returning to a known place retrieves the same image instead of describing it again.
\(\mathcal{S}\)
Dynamic stateWhat changes: a character's location, outfit, visible state, mood, goals, and carried items; a location's time of day, weather, lighting, and props.
\(\mathcal{N}\)
Narrative contextThe main storyline and its milestones, side quests, the current objective, and a turn history of instructions and one-line summaries. Generation prompts are left out so planning stays about the story.
\(\mathcal{G}\)
Place graphConnections between locations, so movement follows known paths and the agent can tell a continuous walk from a jump.
\(\mathcal{T}\)
Segment treeEvery generated segment as a node with its directive, prompt, reference order, frames, continuation link, and state snapshots from before and after.
Branches and shared play
Undo, regenerate, branchUndo restores the state saved before a segment and moves back to its parent; the next turn grows a sibling. Regenerating also adds a sibling. Branches never overwrite each other, and registered assets and place connections are shared by all of them.
Action contracts for each playerIn co-op, each input becomes a structured contract of actor, action, target, path, and end state. Contracts are prepared as soon as a player submits, while the others are still deciding.
One joint shotA director model plans composition, camera, staging, and timing, and a writer drafts the shot with one placeholder per action. The compiler then inserts each locked contract exactly once, bound to its actor.
Forecasts are not factsEach round forecasts a world event and suggests an action for each player. Only what the observation model sees in the keyframes enters the confirmed history.
Less waitingWhile a segment plays, the next joint shot is rendered speculatively for the suggested actions. It is kept only if the submitted round matches it exactly: same head node, shared assets, characters, and action text.
Murea-Agent. (a) The persistent world state. (b) One turn: a single planner call returns the directive and the six-section prompt; the compiler resolves events, motion plan, and state updates and binds every name to a registered asset; the video backend renders by continuation or cut; the result is stored as a node. (c) During playback, the observation model compares the final frame with the planned event and writes back what was seen. Open full size
A journey built from many actions
Longer story paths show each world unfolding across successive generated turns.
Murea turns the agent's conditions into synchronized video and audio. Text specifies the scene, actions, and dialogue; reference images specify appearance; optional voice samples specify timbre. The three parts below address reference fidelity, sampling cost, and temporal continuity.
Report section 3
Omni-Reference Generation
Each reference image is encoded by the video VAE into one clean latent frame. Each voice sample becomes an audio-VAE latent sequence that keeps its timing, rather than a single speaker embedding. References join the target along the token dimension, up to 20 per modality, and are never noised or supervised.
Learned slot IDs tell references apart. In rotary position coordinates, images sit at \(-N_v,\dots,-1\) s before the target and voice samples end at \(-0.04\) s, so they read as context, not as past events. Audio–video cross-attention excludes reference tokens in both directions; reference information reaches the other modality only through the target.
MeanFlow, then distribution matching
Stage one, audiovisual MeanFlow, learns the average velocity over a time interval under a frozen teacher, so one model can sample in a single step or several. Its Jacobian–vector product runs through a custom FlashAttention-4 kernel.
\[x_r^m=x_t^m+\Delta_m\,u_\theta^m(x_t,t,r;c)\]
Stage two, Distribution Matching Distillation, starts from the MeanFlow weights. The score difference between the frozen teacher and an auxiliary denoiser that tracks the student guides it toward better visual detail and motion. The two stages run one after the other.
Streaming with bounded memory
For long rollouts the model extends the stream one interval at a time. Video keeps one preceding chunk. Audio keeps several, so an unfinished sentence and its prosody carry across the boundary instead of restarting or repeating words.
In training, the history comes from ground truth and is deliberately degraded with noise or latent perturbations, preparing the model for the imperfect context it meets at inference. The same MeanFlow-to-DMD recipe extends to streaming with the history added to the conditions.
What the model sees when it generates interval \(k\)
History, degraded during trainingCurrent interval, denoised and supervisedFuture, never visibleReference images and voice samples stay clean throughout.
The generation framework. (a) Omni-reference joint training: clean references, text through cross-attention, flow-matching supervision on valid target entries only. (b) Few-step distillation: AV MeanFlow with a custom FlashAttention-4 JVP kernel, then distribution matching with alternating student and auxiliary updates. (c) Streaming continuation on bounded, degraded histories; at inference, generated intervals fill the history buffer. Video frames and audio illustrate the workflows using validation samples. Open full size
Many styles. Each with its own story.
From painted landscapes to everyday life, explore more worlds in their original story order.
The distilled version of Murea-Pro generates a complete 5.04-second video clip in 3.6 seconds on NVIDIA RTX PRO 6000 Blackwell GPUs—faster than the clip plays.
Report section 4
Murea-Pro · Distilled
3.6 s
End-to-end clip generation
NVIDIA RTX PRO 6000 Blackwell GPUs
Generate faster. Keep the world moving.
Generation time3.6 s
Video duration5.04 s
1.4× real-time speed · 0.71 real-time factor
Timing covers generation of the complete clip, not streaming first-frame latency. WBench quality scores use a separate 50-step configuration.
How we reach real time
Few-step distillation reduces denoising from 50 to 4 steps. Efficient attention, caching, and super-resolution reduce the cost of each step. On eight NVIDIA H20 GPUs, the same optimized pipeline takes a 5.04-second clip from 231.6 s to 6.96 s; on RTX PRO 6000 Blackwell GPUs, it reaches 3.6 s.
Ulysses sequence parallelismSplits the audiovisual sequence across GPUs; all-to-all exchanges let each device attend to the full sequence for a subset of heads.
SageAttention or FlashAttention-4Chosen per GPU architecture. The measured configuration uses SageAttention's low-precision kernels.
Adaptive BlockCache and reference cachingReuses a cached residual for a range of Transformer blocks while hidden-state drift stays small, and reuses VAE encodings of unchanged reference images.
1.5× spatial super-resolutionVideoSharpenSR lets the base generator work at a lower resolution while keeping the delivered output size.
Results
The two generator configurations are evaluated on complementary benchmarks: Murea-Pro on multi-turn interaction, Murea-Fast on reference-conditioned generation. The full agent loop is specified by the system design rather than scored end to end.
Report section 5
Multi-turn interaction on WBench
WBench Full-289 has 289 cases and 1,058 interaction turns covering navigation, subject action, event editing, and perspective switching. Each turn receives the instruction and the final frame of the model's own previous segment, so later turns are judged in the context the model itself produced.
Murea-Pro has the highest Overall score, 1.71 points above Kling 3.0. The lead comes from Interaction (+9.52) and Physical (+5.23). It is within 0.05 of Kling 3.0 on Quality and trails it on Setting (−4.48) and Consistency (−1.70).
Per-metric scores against four baselines
Murea-Pro leads all four on navigation trajectory, event-edit, subject-action, and perspective-switch adherence, and causal fidelity. By task it scores 90.92 on event editing, 87.04 on subject action, 80.65 on perspective switching, and 71.79 on navigation, its weakest task.
30100Bold and underlined: best in the row
Reference-conditioned generation
OpenS2V-Eval's Open-Domain split has 180 cases across seven reference configurations, from a single face to several people with objects. OS2V-Pro is our extended evaluation of the same clips, adding per-reference fidelity, attribute binding, rendering quality, and audio. Murea-Fast uses eight distilled steps and generates 121 frames at 24 fps.
On OpenS2V-Eval, Murea-Fast scores 64.98, the highest among open-source models, with the best NexusScore of any system. Seedance 2.0 scores 67.13 but returned 148 of 180 clips; on those 148 shared cases the gap is 1.02 points. On OS2V-Pro, Murea-Fast scores 69.9, 0.2 below Kling V3 Omni, with the best per-reference fidelity and the joint-best identity consistency.
Murea-Fast is prompted with expanded captions, and OS2V-Pro's task-following (VTF) and binding (Bind) checklists are derived from them, which gives it an input advantage on those two metrics and on the OS2V-Pro aggregate. API systems were queried with the official prompts. Clips a system did not return are not scored as zero, so read results alongside clip counts. Audio quality (ATQ) and event–sound alignment (VESA) are excluded from the aggregate; on both, Murea-Fast trails Kling V3 Omni (59.7 vs 66.3 and 75.0 vs 88.7).
Qualitative results
Each example pairs the input references with frames sampled from the generated video, in temporal order. References range from single images to three-view sheets, and up to five are combined in one clip.
A person, an animal, an object, and a scene in one shot
Three-view reference sheets with a dish and a room
Four faces and a laptop in one group scene
A stylized scene from five references in vertical format
Two characters, two props, one stage across several shots
Data and captions
Multi-reference audiovisual training needs captions that say which reference defines each entity, which voice belongs to each speaker, and when things are said and done.
The dataset combines data created for this work with a curated subset of HuMoSet. Automated filters screen resolution, duration, audio availability, visual quality with DOVER and LAION-Aesthetics, and reference–target identity with ArcFace across sampled frames. Human annotators then judge reference consistency, audio consistency, and video quality.
TransNet finds shot boundaries and speech recognition extracts transcripts with timing. Gemini 3.1 Pro then writes structured caption JSON from the pre- and post-cut frames, the transcript, and the visual and voice references. Each caption binds characters to their voice references, attributes every line of dialogue, and describes each shot with its start time.
Multi-reference audiovisual captioning. Shot boundaries and ASR provide temporal evidence; Gemini integrates frames around each cut, transcripts, and visual and voice references into caption JSON with explicit character–voice bindings and dialogue attribution. Open full size
Team
Core contributors
Linux4,5, Hexin Zhang3,4, Haoshuai Fu1,3,4, North Ge5, Jun Huang1,2, Yuankang Shi1,2, Lingfei Li1,2, Ryan Xu6, Captain6,
Listed by task, with no indication of priority: 1 Agent, 2 Bench, 3 Data, 4 Training, 5 Infra, 6 Project lead. Corresponding author.
Contributors
Bin Chen, Haodong Yuan, Lu Fan, Rui Lan, Shaobin Gao, Xiwei Zheng
Listed alphabetically.
Project sponsors
Jianqi Zhang, Xu Chen
Cite this work
@techreport{murealabs2026inworld,
title = {InWorld: From Imagination to Living Worlds with
Agents and Fast Video Generation},
author = {{Murea Labs}},
institution = {Murea Labs},
type = {Technical Report},
year = {2026},
url = {https://murea-labs.github.io/InWorld/}
}
