---
title: "Understanding video models: Part I - Tokenization"
author: "Anirudha Majumdar (@Majumdar_Ani)"
source_url: "https://x.com/Majumdar_Ani/status/2067619531124506742"
published_at: "2026-06-18T14:44:45.000Z"
fetched_at: "2026-06-19T13:52:52Z"
updated_at: "2026-06-19T13:52:52Z"
language: "en"
review_status: "source-checked"
---
# Understanding video models: Part I - Tokenization
This is the first post in a series I am planning on video models. My goal is to solidify my own understanding of different design choices: tokenization, architectures, data, pre-training, post-training, inference, and evaluation. Hopefully, these distilled notes will be useful to others as well.
### Why do we need tokenization?
Generative video models are based on diffusion-based frameworks (Sohl-Dickstein’15, Song ‘19, Ho ‘20), typically instantiated through flow matching (Albergo ‘22, Lipman ‘22, Liu ‘22). In order to lower computational demands and improve prediction quality, diffusion models are trained in a latent space that compresses low-level perceptual details (Rombach ‘21). By working entirely in this compressed space, latent diffusion models avoid the burden of training and inference on high-resolution image or video data. Intuitively, offloading the reconstruction of low-level details to a separate model allows the diffusion model to focus on generating higher-level semantic features.
### Tokenization: inputs and outputs
The role of a video tokenizer is to provide the compressed latent space in which diffusion modeling is done. For video models, there are two axes of compression: spatial and temporal. Thus, at a high level, a video tokenizer should spatiotemporally compress raw video data, while preserving the ability to reconstruct uncompressed outputs from the latent space.
**Handling both images and videos**
It is desirable to have a unified tokenizer for both images and videos. MAGVIT-v2 (Yu ‘24) suggested structuring inputs to the tokenizer as arrays of shape [1+T, H, W, 3]. Thus, a single image corresponds to T = 0. With this design choice, an input of shape [1+T, H, W, 3] is encoded to a latent representation of shape [1+T', H', W', C], where T/T' is the temporal compression factor (typically 4) and H/H' = W/W' is the spatial compression factor.
**Why not just use .mp4?**
Abstractly, video tokenization has the same goal as classic video compression schemes (e.g., VVC H.266; Bross ‘21). While standard compression methods could be used as tokenizers for video modeling, state-of-the-art neural network-based tokenizers have improved compression-quality tradeoffs; see Figure 6 of the MAGVIT-v2 paper. One might then ask why video compression has not been taken over by neural tokenizers. The reason for this is computational — standard compression schemes run efficiently on CPUs, which is critical for their broad use cases. However, neural tokenizers require GPUs/TPUs for efficient implementation.
### Tokenization through VAEs
Video tokenization today is almost exclusively based on variational autoencoders (VAEs; Kingma ‘13). The most widely used tokenizers include the WAN-VAE (WAN team ‘25), Stable Diffusion VAE, COSMOS tokenizer (NVIDIA ‘25), and MAGVIT-v2 (Yu ‘24). Below, I discuss key design choices in different tokenization schemes.
**Discrete vs. continuous**
The first major fork is between continuous and discrete tokenizers. Continuous tokenizers encode a video into a continuous latent representation:
ℰ : ℝ^(1+T×H×W×3) → ℝ^(1+T′×H′×W′×C)
This enables the training of diffusion models in this compact latent space; see, e.g., WAN (WAN team ‘25).
Discrete tokenizers consist of an encoder and a quantizer:
ℰ : ℝ^(1+T×H×W×3) → ℝ^(N×K), 𝒬 : ℝ^K → 𝒮 (finite codebook)
Here, the encoder produces a length-N sequence of latent variables of dimension K. The quantizer then maps each latent variable in the sequence into a discrete token belonging to a finite codebook. In order to decode tokens into videos, discrete tokens are de-quantized and then passed through a decoder:
𝒟 : ℝ^(N×K) → ℝ^(1+T×H×W×3)
Discrete tokenizers are based on variants of VQ-VAEs (van den Oord ‘17). In particular, MAGVIT-v2 (Yu ‘24) introduced a variant of the VQ-VAE, which they refer to as lookup-free quantization (LFQ). By eliminating the embedding dimension in the VQ-VAE codebook, LFQ avoids the computation of the closest point in the finite codebook. VideoPoet (Kondratyuk ‘24) also uses the MAGVIT-v2 tokenizer. The NVIDIA COSMOS tokenizer (NVIDIA ‘25) first transforms videos into a wavelet representation and then uses FSQ (Mentzer ‘23) for quantization.
In contrast to continuous tokenizers, discrete tokenizers enable the use of language modeling architectures for video generation. In principle, this allows one to leverage all the advancements in language modeling (e.g., architectural improvements, training and inference optimization, etc.), while also providing a unified representation for video generation (videos out) and video understanding (videos in).
### Understanding the interplay between compression, reconstruction quality, and video generation quality
There is a delicate interplay between the amount of compression performed by a video tokenizer, the quality of reconstruction, and the quality of videos generated by training on tokenized representations. The tradeoff between compression and reconstruction quality is intuitive: higher compression rates make it harder to preserve details necessary for high-fidelity reconstruction.
The interplay between compression and video generation quality is less intuitive. One might think that initially, video modeling becomes easier with more compression, but then video quality degrades if representations are too compressed (i.e., there is a “sweet spot” for compression). As shown in the figure below from the MAGVIT-v2 paper, this is indeed the case for a VQ-VAE tokenizer. However, using LFQ, MAGVIT-v2 demonstrates that is it possible to increase the vocabulary size (i.e., lower compression) while *simultaneously* improving reconstruction *and* generation quality (at last up to the vocabulary size considered in the figure).
More recent work (Yao ‘25) has found that generation and reconstruction quality can also be improved by aligning features of the tokenizer to pre-trained vision foundation models.
### Architecture: causal tokenization
State-of-the-art video tokenizers have all converged toward *causal* architectures for tokenization (WAN tokenizer, MAGVIT-v2, COSMOS): future frames do not influence the representations of past frames. As an example, the NVIDIA COSMOS tokenizer (NVIDIA ‘25) first processes groups of four frames through a wavelet transformation and then applies causal convolution and attention layers.
More recently, the COSMOS models have adopted the WAN tokenizer (see COSMOS3; NVIDIA ‘26), which also employs a causal architecture that is similar to MAGVIT-v2.
There are actually a number of reasons for using a causal architecture in video tokenization, which are a bit more subtle than I had initially appreciated.
**A non-argument for causal tokenization.** First, in order to understand the subtlety, let’s consider a case where causal tokenization is actually *not* necessary. Suppose that you want to train a full-sequence diffusion model (i.e., not auto-regressive) which will *only* be used in text-to-video mode (text in, videos out). Here, a non-causal architecture is actually fine. In fact, it might be beneficial to have features from future frames inform past frames (e.g., when something becomes clear in the future in a way that can improve the quality of generation in the past).
**I2V or V2V generation.** Things become more interesting in image-to-video (I2V) or video-to-video (V2V) generation. Here, causal encoding and decoding is quite convenient. A causal tokenizer encodes a given image/video context x into z and the video model generates future encoded frames z'. The decoder takes z and z' and reconstructs future frames x'. Making a non-causal scheme work is much trickier. For example, if training sequences (x, x') are encoded through a non-causal encoder, the representation of past frames x depends on the representation of future frames x'. But, at inference time, we don’t have access to future frames, and so cannot compute the correct encoding for the conditioning frames x.
**Inference speed.** For streaming video generation, causal tokenization is again very convenient. Once z' (encoded future frames) are generated, these can be used as conditioning for generating more frames without having to recompute the representation. Section 4.1.3 of the WAN tech report (WAN team ‘25) has additional discussion on how causal tokenization enables feature caching for efficient inference.
### Loss functions and training curriculum
Training for video tokenizers is performed in multiple stages.
**Image tokenization.** For the WAN model, the first stage trains a VAE for images only (T = 0). This provides a spatial prior, which is useful for training the full video tokenizer.
**Low-resolution training.** In the second stage, the image VAE is “inflated” to have a temporal component and then fine-tuned on low-resolution (128 x 128) frames with short video sequences (5 seconds). In this stage, training is performed with a combination of L1 reconstruction loss, the VAE KL prior loss, and LPIPS (for encouraging perceptual similarity; Zhang ‘18).
**High-resolution training.** In the third stage, the tokenizer is fine-tuned on higher quality videos with different resolutions and video lengths. In this stage, an adversarial loss is introduced following VQ-GAN (Esser ‘21).
### Emerging methods: adaptive, hierarchical, and multi-view tokenization
Video tokenization is an active area of research. Below, I discuss some exciting lines of work.
**Adaptive tokenization.** Standard video tokenizers uniformly allocate tokens to different parts of the video. However, some parts of the video are significantly more complex than others. A recent line of work on *adaptive tokenization* seeks to allocate more tokens to “more interesting” parts of the video. ElasticTok (Yan ‘24) does this by randomly masking some number of tokens during training, forcing the other tokens to represent information in the video. InfoTok (Ye ‘25) uses a fixed-compression tokenizer (e.g., the COSMOS tokenizer in their experiments) to ascertain how many tokens are necessary for a given video by predicting the reconstruction loss. This token budget is then used by an adaptive compressor which eliminates tokens with the lowest information content.
**Tokenization for long-horizon generation.** Another emerging line of work is on *hierarchical* tokenization. MilliVid (Chandratreya ‘26) encodes videos into a hierarchy of tokens that capture different levels of coarse-to-fine information. This enables improved consistency and quality in long-horizon video generation. FramePack (Zhang ‘25) has a similar motivation of long-horizon generation, but allocates more tokens to “more relevant” past frames (e.g., based on temporal proximity or feature similarity).
**Multi-view tokenization.** SceneTok (Asim ‘26) encodes multi-view scene observations into tokens. These serve as diffusable representations for scenes, which can be used to generate videos from novel views.
### Do we really need video tokenization?
Let’s return to the argument for video tokenization we started this post with. Tokenizers provide a compact latent representation, which makes diffusion modeling more efficient in terms of computation and data. However, the tokenizer also imposes a hard bottleneck; decoded videos can only capture what the latent space is capable of representing. If the tokenizer discards some information, generated videos will have artifacts that cannot be eliminated by the diffusion model.
Why delegate the representation of fine details to the tokenizer? Why can’t the diffusion model be responsible for everything? With this motivation, we are seeing a line of work that bypasses tokenizers and works in the raw image space (see, e.g., JiT and AsymFlow). These methods are starting to be competitive with latent diffusion models.
My personal bet is that tokenizers are a short-term crutch, and that eventually we will find ways to unshackle the diffusion model from the bottleneck of frozen tokenizers.