Video Generation - How It Works

Video generation models extend the same diffusion process used for image generation, but with one critical addition: temporal coherence - the model must generate frames that flow naturally from one to the next, maintaining consistent physics, objects, and lighting across hundreds or thousands of frames. This is significantly harder than generating a single image, which is why video generation lagged image generation by several years.

From Image Generation to Video

Image diffusion models (Stable Diffusion, DALL-E 3) work in a 2D latent space: the model learns to denoise a 2D grid of latent vectors into a coherent image. Video generation extends this to a 3D latent space - height × width × time:

Image generation:
  Noisy 2D latent (H × W) → Denoised image

Video generation:
  Noisy 3D latent (H × W × T) → Denoised video (T frames)

  T = number of frames (e.g., 25 frames @ 25fps = 1 second)
  The model must denoise ALL frames simultaneously,
  maintaining consistency across time.

Temporal Attention - The Key Innovation

Standard transformer attention allows each token to attend to all other tokens within the same frame. Temporal attention extends this across frames: each spatial position can attend to the same spatial position in adjacent frames. This is what maintains object consistency - the model "knows" that the red ball at frame 10 is the same red ball from frame 1:

Spatial attention: token at (x=5, y=3, t=10) attends to
                   all tokens at t=10 within the frame

Temporal attention: token at (x=5, y=3, t=10) attends to
                    tokens at (x=5, y=3) across all t=1..T
                    → Maintains spatial position consistency
                       across time

3D-full attention (attending to all positions across all frames simultaneously) is more powerful but computationally expensive - it scales as O(H × W × T)². Most models use factored attention (spatial + temporal separately) to make this tractable.

DiT - The Architecture Powering Sora

Earlier video models used U-Net backbones (modified from image diffusion). The architectural shift enabling higher quality: Diffusion Transformers (DiT) - replacing the U-Net with a pure transformer architecture. Sora is built on this principle.

Why DiT for video:

• Transformers scale more predictably than U-Nets - larger models consistently produce better quality
• Attention can flexibly model long-range dependencies across frames - critical for maintaining consistency in long videos
• Patches-based: the video is divided into spatiotemporal patches (video tokens), processed uniformly - enabling variable resolution and duration without architecture changes

Full Generation Pipeline

Text prompt / Image
       │
       ▼
Text Encoder (T5 / CLIP)
  → Encodes the prompt into condition embeddings
       │
       ▼
VAE Encoder (for image-to-video)
  → Encodes the input image/video into latent space
       │
       ▼
Noisy 3D Latent (Gaussian noise)
       │
       ▼  ← repeated N times (typically 25-50 denoising steps)
DiT / 3D U-Net
  → Predicts noise to remove at this step
  → Uses text condition embeddings via cross-attention
  → Uses temporal attention for frame consistency
       │
       ▼
Denoised 3D Latent
       │
       ▼
VAE Decoder
  → Converts latent → pixel frames
       │
       ▼
Video (MP4) - H × W × T frames

Practical Constraints

• Cost - video generation is 10–100× more computationally expensive than image generation. A 10-second 720p video can cost $0.50–$5.00 via API.
• Latency - generation takes 30 seconds to several minutes. Not suitable for real-time applications.
• Duration–quality trade-off - longer videos require more memory. Current models cap out at 30–120 seconds; quality degrades in longer clips.
• Character consistency - maintaining a specific character's appearance across multiple separate generation calls remains difficult. Each generation starts fresh.
• Physics accuracy - models have learned statistical patterns from video data, not actual physics. Fluid dynamics, small object interactions, and hands are often wrong.

Image-to-Video

Most video generation models support image-to-video: provide a still image as the first frame, then generate the video continuing from that frame. This is often more useful than pure text-to-video because:

• The starting frame is deterministic and controllable
• You can generate the starting image with a precise image generation model, then animate it
• Character and scene consistency is naturally maintained from the reference frame