STAGING DRAFT · numbers and copy under review · do not share

WMEval

Interactive world models, scored against a simulator engine twin: Unity runs the same actions and logs the true state.

Lanxiang Hu^1*, Yixin Huang^1*, Abhilash Shankarampeta^1*, Zilin Dai^1*, Yuyuan Wu^1*, Kun Qian^2*, Junze Liu^2*, Arvind Srinivasan^2*, Mingjia Huo¹, Will Lin¹, Kai Zhong², Weiqi Zhang², Tian Wang², Qingjun Cui², Tajana Rosing¹, Hao Zhang¹

¹UC San Diego · ²Unity · ^*Main contributors

Paper Code Dataset Results BibTeX

World models look right frame by frame. Next to their engine twin, the cracks show: actions that don't take effect, physics that breaks cause and effect, and scenes that don't stay consistent.

TL;DR

Every rollout has an engine twin. The same action trajectory runs in a Unity scene and in the world model. The twin supplies frames and true state: poses, velocities, masses, momentum, lighting and causal events.
Engine twins are where world models are heading. Code world models, sim-to-real video renderers and synthetic-data pipelines all pair generated video with an engine. Measuring how much true state survives into pixels becomes the core question, and WMEval measures it today.
7 dimensions: object & character consistency, controllability, kinematics & mechanics, fluids, causality, lighting and prompt consistency, each scored by evaluators calibrated on the twin itself.
6 world models, three gaps. Four from the paper, plus LingBot-World v2 (Cam) and Odyssey scored with WMEval Lite. Following a control ≠ evolving the world correctly. Looking similar ≠ being physically right. Local consistency ≠ global consistency.

Ranking

Leaderboard

Each dimension is ranked on one metric, scene by scene: a model's dimension rank orders the models by their average rank across the dimension's scenes. The overall score averages, over all dimensions, the share of the other models a model ranks above, and models are ordered by that exact average, so it combines ranks, never raw scores across metrics.

Rank	Model	Score ↑	Control	Consistency	Kinematics	Fluid	Lighting	Prompt
1	Genie 3	0.767	1/6	1/6	1/6	2/6	4/6	1/6
2	LingBot-World (Act)	0.467	3/6	3/6	3/6 ‡	3/6 ‡	2/6	1/6
2	Odyssey	0.467	4/6	1/6	3/6	1/6	5/6	3/6
4	Matrix-Game 2.0	0.433	1/6	4/6	2/6	3/6 ‡	3/6	6/6 ‡
5	LingBot-World v2 (Cam)	0.267	5/6 ‡	5/6	3/6 ‡	3/6 ‡	1/6	3/6
6	Waypoint-1.5	0.033	5/6 ‡	6/6 ‡	3/6 ‡	3/6 ‡	6/6	5/6

Cells give rank / models. ‡ marks a dimension the model's interface can't express: it is ranked tied last and scores 0 there. Ranks come from exact averages, so small differences within seed-to-seed noise still order models. Each dimension is ranked on one metric: Control, camera-compensated flow cosine against the Unity reference (does the controlled object, not the camera, move as commanded); Consistency, character consistency (DINOv2 identity); Kinematics, final displacement error; Fluid, surface-tilt error; Lighting, multi-object sun disagreement (do the shadows of different objects agree on one sun); Prompt, overall rubric score. Causality and cLPIPS are not included in the leaderboard. HY-WorldPlay and Matrix-Game 3.0 are left out. Kinematics is ranked on RacingCar only (third person), with trajectories measured against the road. Control is measured with WMEval Lite on new videos for every model (five seeds per scene), after removing each video's camera motion. Odyssey and LingBot-World v2 (Cam) are scored with WMEval Lite, five seeds per case.

Dimension	Ranking metric	Rank · models	Not ranked
Control	Camera-compensated cosine	1 Genie 3 1 Matrix-Game 2.0 3 LingBot-World (Act) 4 Odyssey	LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡
Consistency	Character consistency	1 Genie 3 1 Odyssey 3 LingBot-World (Act) 4 Matrix-Game 2.0 5 LingBot-World v2 (Cam)	Waypoint-1.5 ‡
Kinematics	Final displacement error (RacingCar only)	1 Genie 3 2 Matrix-Game 2.0	LingBot-World (Act) ‡, Odyssey, LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡
Fluid	Surface-tilt error	1 Odyssey 2 Genie 3	LingBot-World (Act) ‡, Matrix-Game 2.0 ‡, LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡
Lighting	Sun disagreement	1 LingBot-World v2 (Cam) 2 LingBot-World (Act) 3 Matrix-Game 2.0 4 Genie 3 5 Odyssey 6 Waypoint-1.5	—
Prompt	Overall rubric score	1 Genie 3 1 LingBot-World (Act) 3 Odyssey 3 LingBot-World v2 (Cam) 5 Waypoint-1.5	Matrix-Game 2.0 ‡

Models are ordered by their average rank across the dimension's scenes; equal averages share a rank. Unsupported models are ranked tied last in that dimension.

Motivation

Looks right ≠ works right

Video generative models are increasingly pitched as interactive world-modeling engines. Given observations and actions, they should produce futures that are both visually plausible and physically coherent. That matters for robotics, game engines, and digital twins. Yet most evaluations still ask whether a generation looks right: visual quality, prompt alignment, or perceptual similarity to a reference clip.

An interactive world model also has to keep objects where they are while the camera moves, respond to the control it was given, conserve momentum through a collision, and carry an action through a multi-stage chain of consequences. Checking these properties requires a reference that knows the true state of the world, and a physics engine is exactly that.

We use Unity as that reference: a widely used real-time engine whose mature tooling and asset ecosystem make diverse, controllable scenes practical. Unity isn't the competitor. It's the engine twin that knows what the right answer was.

Benchmark category	Representative works	Perceptual quality	Prompt consistency	Physical-law adherence	Action controllability	Causality
Video model benchmarks	VBench, VBench-2.0, Video-Bench	✓	✓	✗	✗	✗
World-modeling video benchmarks	VideoScience-Bench, WorldModelBench	✓	✓	✓	✗	✗
Interactive world-model benchmarks	MIND	✓	✗	✗	✓	✗
Interactive, simulator-grounded	WMEval (ours)	✓	✓	✓	✓	✓

✓ explicitly evaluated · ✗ not a primary evaluation target. Coverage as summarized in the paper.

Why engine twins

The engine twin is the asset

A simulator reference can look like overhead: build scenes, script actions, export state. We see it the other way around. A twin turns "does this video look plausible?" into "how far is the recovered state from the true state?", which has an exact answer.

01 · EXACT

Ground truth by construction

Every frame of the twin carries poses, transforms, masses, velocities, lighting and event labels. No annotation, no guessing what should have happened.

02 · CALIBRATED

Measurement you can audit

Run the same reconstruction on the twin's own render and you get the evaluator's error floor. Model error is what remains above it.

03 · SCALABLE

More scenes, same contract

New scenes and trajectories add cases without new labels. WMEval ships them as Unity-free bundles: frame, actions, reference and state.

Where this is heading: code world models. A growing line of work writes the world as executable scene and physics code, runs it in an engine, and renders it with a video model. There every rollout has a twin by design, and evaluation splits cleanly: is the engine state right, and how much of it survives into the pixels? The second question is exactly the state-from-video delta WMEval measures today, with the same twins, reconstruction pipelines and calibration.

Protocol

Same actions, two worlds

Every WMEval case pairs a Unity scene with a deliberately designed action trajectory. Unity renders the reference rollout and logs its internal state. The world model sees only the conditioning frame, the prompt, and the same actions, and its state stays latent. Evaluators then recover the matching quantities from the generated video and compare.

01 · DEFINE

Semantic action trajectory

Per scene and dimension: actor, camera, steering, or object controls in [-1,1] plus discrete branches, timed in physical seconds. Controllability trajectories induce clear motion, mechanics trajectories trigger collisions, and causality trajectories trigger multi-stage chains.

02 · EXECUTE

Matched rollouts

Unity returns the observation ô, the state ŝ, and metadata m̂ (velocities, trajectories, lighting, event labels). Each world model gets a lightweight adapter that maps the semantic trajectory onto its native keyboard, mouse, or camera interface and aligns the timing.

03 · MEASURE

Recover and compare

CV pipelines (RAFT, GroundingDINO, SAM 2, DINOv2, depth) and VLM judges recover physical quantities from the generation. These are compared with Unity's ground truth, alongside event-matched LPIPS and rubric-based prompt consistency.

WMEval pipeline: action trajectories drive both Unity and world models; Unity states and CV-recovered world-model states feed dimension-specific and holistic metrics — **WMEval overview.** Predefined or synthetic trajectories drive actor and camera controls. Unity produces observations plus states (velocity, lighting, momentum, and more). For a world model, those states are recovered from its observations with CV-based techniques. Dimension-specific and holistic metrics compare the two.

Only fair comparisons. World models expose very different control interfaces, so each model is scored only on the dimensions its interface can express. A first-person keyboard-only model is not graded on third-person object control, and an unsupported control is reported as N/A, never as a zero.

Benchmark tour

Nine scenes, one stress test each

These are Unity reference rollouts. Each clip plays when you scroll it into view. Tags show which dimensions each scene probes.

Side by side

Engine twin vs. world model

Each clip pairs the Unity ground-truth rollout with a world-model rollout (Genie 3, LingBot-World, Odyssey or Matrix-Game 3.0) driven by the same semantic action trajectory and started from the same conditioning frame. Unity frames are matched to the commanded camera motion the same way the evaluator matches them. In the Genie clips, the on-screen WASD overlay shows the keys actually issued.

Pair

0.0s

What we measure

Evaluation dimensions

Each dimension compares measurements recovered from the candidate video against quantities derived from Unity. Holistic metrics then add video-level perceptual similarity and prompt-scene consistency.

Key learnings

What the engine twin reveals

Three gaps recur across dimensions. Each is invisible to perceptual metrics and obvious against the twin's state.

Gap A · Acting ≠ evolving

Control ≠ correct dynamics

On MirrorRollingBall, Genie 3 and LingBot-World move in the commanded direction almost perfectly. Their subsequent state evolution still deviates substantially from Unity's optical flow. BeamDomino shows the same pattern (cosine 0.99, state error 0.44 / 0.34).

0.98 / 1.00commanded-direction cosine

0.77 / 0.49relative flow error vs. Unity

See controllability table →

Steps don't compose into chains

On BeamDomino (rolling → impact → topple → cascade), Genie 3 answers isolated causal questions almost as well as the reference (ICRA 90% vs. 100%), yet preserves the full chain in only 3 of 15 rollouts. None of its 10 glancing or direct-strike rollouts completes the chain.

13/15Unity (judge ceiling)

3/15Genie 3

See causal-chain frames →

Gap B · Looks similar ≠ is right

LPIPS hides interaction failures

Genie 3's event-matched LPIPS on BeamDomino is nearly identical to LingBot-World's (0.60 vs. 0.61), yet it completes the causal chain in 3/15 rollouts versus 13/15 for Unity. On BallRoller, Genie moves against the commanded direction (cosine −0.33), and LPIPS does not show that the motion is wrong.

0.60 ≈ 0.61causal LPIPS

3/15 vs 13/15full chains (Genie vs. Unity)

See causal LPIPS →

Better LPIPS, not better physics

On LiquidContainer, HY-WorldPlay has a far lower LPIPS than Genie 3 (0.300 vs. 0.624), yet its surface orientation deviates no less from Unity. It renders the bottle at about half the reference size, so a well-matched background dominates the perceptual score.

0.300 vs 0.624LPIPS

9.62° vs 8.87°surface tilt error

See fluid table →

Gap C · Local ≠ global

Foreground holds, scene drifts

On third-person RacingCar, Genie 3, LingBot-World, and HY-WorldPlay all score a perfect 1.00 on the car and only 0.71–0.76 on the environment. The VLM judge agrees with human ratings (Spearman ρ = 0.90).

1.00foreground objects

0.71–0.76environment

See prompt consistency →

Big camera moves break layout

Under perturbation and 360° orbits in WanderingGarden, anchor objects are re-found in only a small fraction of frames for every model. The MirrorRollingBall character stays recognizable (99.9% for Genie under perturbation), while the pairwise geometry around it drifts by 25–37%.

0–31%Garden object permanence

25–37%Mirror pairwise-distance error (Genie)

See consistency explorer →

Numbers

Results explorer

No model leads every dimension: rankings change between consistency, physics, lighting and prompt adherence. Good perceptual scores often sit beside large physical or causal errors. Expand for the full per-dimension tables.

Mean ± std over five generations per cell unless noted. ↑ higher is better, ↓ lower is better. Best model per column is highlighted. “—” and N/A mark interfaces or views outside a model's supported protocol, not failures.

Visual evidence

Failures, frame by frame

World models render each stage plausibly yet break what follows: cascades stall, bottles vanish, shadows disagree with the sun. These frames show the failures behind the numbers.

BeamDomino canonical trajectory, frames at rolling, impact, topple, cascade, terminal for Unity and Genie 3 — **Physical causality, BeamDomino (canonical).** Key frames at each causal stage. In the Unity reference the strike topples the support and releases the cascade. A world model can render each stage plausibly and still fail to propagate the consequence to the next one.

BeamDomino direct-strike trajectory for Unity and Genie 3 — **Physical causality, BeamDomino (direct strike).** Genie 3 preserved the full chain in none of its glancing or direct-strike rollouts.

Fluid mechanics, LiquidContainer (chirp-strafe profile). The Unity row gives the reference slosh; Genie 3 drifts to a different scene and loses the bottle. All physics axes are computed inside a SAM 2 liquid mask whose error is calibrated against Unity.

SunRisesAndSets snapshots at four orbit fractions for Genie 3, LingBot-World, Matrix-Game 2.0, Matrix-Game 3.0 — **Lighting and shading, SunRisesAndSets.** Rows from top: Genie 3, LingBot-World, Matrix-Game 2.0, Matrix-Game 3.0. Columns are 0/33/66/99% of a fixed-sun walking orbit. Some rollouts leave the scene entirely (low Scene Retention). Others keep it in view while shadows point in inconsistent directions.

Matrix-Game 3.0 scene dropout frames — **Scene dropout.** A Matrix-Game 3.0 rollout pans off the table at frame 59 and renders empty floor and sky for the rest of the orbit.

Trusting the measurements

Evaluator calibration

Every evaluator is first scored on Unity's own rollouts, so its error floor and judge ceiling are known; where the floor rivals model error (WanderingGarden depth), differences are read qualitatively. Conclusions hold across run subsets, and the prompt judge agrees with humans (ρ = 0.90).

Calibrated against the simulator. Learned evaluators (trackers, depth, segmentation, flow) are first run on Unity's own rollouts, where the true quantities are known. This tells us their error, and we compare models only within each metric's validated scope. A metric that cannot tell Unity's own rollout from a wrong one is replaced: the RacingCar trajectory is now measured against the road, and a wrong path scores worse in 99.6% of pairs, up from 48% when the car was projected through Unity's camera.
The reference gets judged too. For VLM-judged causality, the Unity rollout goes through the same judge. Unity's 13/15 is a measurement ceiling, not a perfect score, and model rows are reported as deltas from it.
Robust to the choice of generations. Main comparisons hold across all C(5,3) = 10 three-run subsets, with metric-appropriate significance tests. When evidence doesn't separate two models (for example, Genie 3 vs. LingBot-World on prompt consistency, p = 0.99), we call them comparable.
Human-aligned judges. The rubric-based prompt-consistency judge agrees with human raters at Spearman ρ = 0.90.

Evaluate your model

Bring your own model

WMEval packages each case as a checksum-locked bundle. A bundle holds the conditioning frame, the semantic action script, the Unity reference rollout, and the simulator quantities each metric needs, so you don't need Unity installed to evaluate a new model. Generate rollouts from the supplied conditions, drop them into a folder, and run one command.

# 0. install (macOS arm64 lock shown; CUDA pins in tools/run_wmeval_lite_modal.py)
git clone https://github.com/hao-ai-lab/WMEval && cd WMEval
uv venv --python 3.11 .venv && uv pip install -r requirements-macos.lock

# 1. download the case bundles from Hugging Face (huggingface.co/datasets/lmgame/WMEval-Lite), checksum-verified
python tools/sync_lite_data.py download              # add --split full for the 220 synthetic trajectories

#    validate the suite and list case × dimension wiring
python tools/run_wmeval_lite.py validate   --suite eval_sets/wmeval_lite_v1
python tools/run_wmeval_lite.py list-cases --suite eval_sets/wmeval_lite_v1

# 2. generate: one video per case/seed from the bundle's frame + actions
#    runs/<model>/<Scene>/<variant>/<seed>.mp4

# 3. evaluate every dimension your model's interface supports (Table 4/7 cells included)
python tools/run_wmeval_lite.py run \
  --suite eval_sets/wmeval_lite_v1 \
  --model-root runs/<model> \
  --output-dir reports/wmeval_lite/<model>

# 4. optional: compare the Table 4/7 cells with the paper baselines
python tools/export_lite_table_results.py --model "<name>" \
  --results reports/wmeval_lite/<model>/per_case_results.jsonl \
  --output  reports/wmeval_lite/<model>/table_results.json

Release in progress. The dataset is public on Hugging Face (lmgame/WMEval-Lite), pinned to a versioned revision. A model-neutral generation adapter template and a locked full-evaluation environment are being finalized. Some dimensions also need GPUs, CV checkpoints, or a hosted VLM judge (API key). Those requirements are separate from Unity, which you never need to install.

Citation

BibTeX

@article{hu2026wmeval,
  title   = {WMEval: Evaluating Interactive World Models Grounded on Physical Simulations},
  author  = {Hu, Lanxiang and Huang, Yixin and Shankarampeta, Abhilash and Dai, Zilin and
             Wu, Yuyuan and Qian, Kun and Liu, Junze and Srinivasan, Arvind and Huo, Mingjia and
             Lin, Will and Zhong, Kai and Zhang, Weiqi and Wang, Tian and Cui, Qingjun and
             Rosing, Tajana and Zhang, Hao},
  journal = {arXiv preprint arXiv:2610.33894},
  year    = {2026}
}