MMDiT for Video

Key Insight

A plain video DiT lets the video tokens read the text prompt through a one-way cross-attention step — the text influences the video, but never the reverse. MMDiT (Multi-Modal Diffusion Transformer) — the SD3 and Flux design — instead sends text tokens and video tokens through the same attention layers, so the two modalities see and shape each other inside one shared operation. That two-way conversation is what helps the model get compositional prompts right ("a red cube on a blue sphere"), and building it for video lets you measure the payoff directly: text adherence should visibly improve over a cross-attention baseline.