A mechanistic look at motion planning
Unveiling the Flaws in Attention Mechanisms
Physical failures can begin when attention commits to the wrong positions.
Relaxing spatial RoPE helps the model explore better motion trajectories.
Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the “motion planning” process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the “first shape, then details” finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.
01 / A simple motion, a revealing failure
The same prompt, four random seeds. Compare the original model with our method.
“Against a pure white background, a basketball falls vertically from mid-air onto a wooden floor and bounces up several times.”
02 / From mechanism to intervention
Motion planning brings text semantics into the video latent and determines where an object appears across frames. In Wan2.1-T2V-1.3B, this happens largely within the first 5 of 50 denoising steps. Cross-attention reveals several candidate positions before a trajectory takes shape; causal interventions distinguish the heads that drive motion from those that merely display its pattern.
Self-attention coordinates video tokens across frames. But 3D RoPE induces a spatial anchoring bias: a query tends to attend to nearby spatial coordinates in other frames. A physically plausible candidate farther away can therefore receive too little attention.
Candidate regions initially compete without a stable winner. When a few frames settle early, spatial anchoring increases support for nearby positions in other frames—even if those positions violate physics. In the seed-20 failure, frame 14 switches from a lower candidate to a higher one as frame 10 stabilizes, leading to a basketball suspended in mid-air.
We reduce the height and width RoPE frequencies during early denoising, weakening excessive spatial decay while preserving temporal RoPE. Section 6.1 applies the following transformation to queries and keys alike:
For one 2D channel pair per axis, q = [qf, qh, qw] ∈ ℂ³ represents six real query channels; p = (pf, ph, pw) ∈ ℕ³ gives the frame, height and width coordinates. Each complex scalar qa encodes a real 2D pair. θ ∈ ℝ is its rotary frequency, i is the imaginary unit, and 0 < λh, λw < 1 are scalar spatial scales. The transformation applies pairwise across all channels.
03 / Beyond basketball
Six examples of everyday physical interactions. The original model appears on the left; our method appears on the right.
Our systematic evaluation uses 344 VideoPhy cases and human judgments of semantic adherence and physical commonsense. The largest gains are on interactions involving solids. See the paper for model-specific settings and quantitative results.
