Reading notes on:

Long-video training fundamentally breaks a simple assumption that has quietly underpinned most distributed training systems: one sample fits on one GPU. When a single video reaches hundreds or thousands of frames—producing hundreds of thousands or even millions of tokens—data parallelism, tensor parallelism, and pipeline parallelism are all insufficient on their own. The bottleneck moves inside the sample: the temporal/context dimension itself. This blog traces how two NVIDIA systems, LongVILA (understanding) and LongLive-2.0 (generation), solve this with two distinct flavors of Sequence Parallelism (SP).

1. Why Video Needs a New Parallelism Axis

The standard parallelism playbook covers three orthogonal dimensions:

Parallelism What it splits Why not enough for long video
Data Parallelism / FSDP / ZeRO Batch, parameters, gradients, optimizer state A single video sample can still be too long for one rank
Tensor Parallelism Hidden dimensions, attention heads Sequence activations may still be too large
Model / Pipeline Parallelism Transformer layers Each stage still sees the full sequence

Sequence Parallelism (SP) adds a fourth axis: partitioning the context, time, or token dimension of one sample across ranks. The mental model is clean:

Short video training:  many samples  → split the batch     (Data Parallelism)
Long video training:   one sample    → split inside sample  (Sequence Parallelism)

But “just split the sequence” is deceptively simple. For video, the partition must respect token origins (where frames become visual tokens), prediction targets (which tokens carry loss), attention masks (causal structure), and hardware topology (NVLink vs. cross-node bandwidth). Generic text-centric SP systems provide the vocabulary but not the final answer.

2. A Map of SP Systems

Before diving into the video-specific designs, it helps to place the key SP systems on the map:

  • Sequence Parallelism (Li et al., 2021): Framed SP as breaking input sequence length limitations by distributing chunks across devices.
  • Megatron SP / Context Parallelism: Megatron-style SP reduces activation memory and interacts naturally with tensor parallelism. Megatron Core’s later Context Parallelism generalizes this by partitioning the sequence dimension for network inputs and activations.
  • DeepSpeed-Ulysses: Partitions input data along the sequence dimension, using all-to-all communication during attention. Efficient when the attention head count supports the required partitioning.
  • Ring Attention: Uses blockwise attention and ring communication of KV blocks—devices stream KV chunks while computing local attention. Context length scales linearly with device count.
  • USP: Unifies Ulysses-style and Ring-style approaches into a broader SP design space.
  • LoongTrain: Pushes toward 2D-Attention and head-context parallelism for long-sequence LLM training.

These systems solve the mechanism of splitting sequences. Long video adds another layer: the sequence is produced by a multimodal pipeline (vision encoder → visual tokens → LLM) or a structured generation objective (clean history + noisy target). Video SP must respect token origin and token meaning, not just token position.

3. SP for Long-Video Understanding: LongVILA’s Multi-Modal SP

LongVILA is a long-context Visual Language Model (VLM) for video understanding. It extends VILA from 8 to 2048 video frames and reports 99.8% accuracy on a 6000-frame needle-in-a-haystack evaluation where the video exceeds 1M tokens. The system contribution is Multi-Modal Sequence Parallelism (MM-SP).

Why Text-Only SP Fails for VLMs

In a VLM, the model starts with raw frames and text, and a vision encoder produces visual tokens before the LLM sees them. Ring-style or text-centric SP can shard the final token sequence, but this ignores the upstream vision encoder workload. If the vision tower processes all frames on one rank while other ranks wait for the resulting visual tokens, the system has a severe load imbalance before attention even begins.

Two-Stage Sharding

MM-SP addresses this with a two-stage strategy:

  1. Stage 1 — Shard by images/frames: Distribute input frames across SP ranks to balance vision encoder workload. Each rank encodes its local frames through the vision tower.
  2. Stage 2 — Shard by tokens: After visual embeddings and text are assembled, rebalance the full token sequence across ranks for the LLM.

This shifts the SP boundary earlier in the pipeline—balancing starts when video becomes visual work, not when the LLM sees a long token sequence.

Topology-Aware Communication

LongVILA contrasts Ring-style SP (point-to-point communication everywhere) with MM-SP’s 2D-Attention design:

  • Intra-node: All-to-All communication over fast NVLink bandwidth.
  • Inter-node: Point-to-point communication handles slower cross-node paths.

This hardware-aware communication layout is not an afterthought—it is a core design decision that determines whether the system can scale beyond a single node.

Extension to Reinforcement Learning

LongVILA-R1 extends MM-SP to RL via Multi-modal Reinforcement Sequence Parallelism (MR-SP):

  • Splits video-frame encoding across GPUs during rollout.
  • Gathers and caches video embeddings for reuse across repeated rollouts.
  • Applies SP to the long video prefix for both policy and reference models.
  • Reports up to 2.1× speedup on 512-frame RL training and scales to 1024 frames without OOM on a single 8×A100 node.

LongVILA takeaway: For long-video understanding, SP must become multi-modal SP. The system must know where visual tokens come from, not only where transformer tokens go.

4. SP for Long-Video Generation: LongLive-2.0’s Balanced SP

LongLive-2.0 tackles long-video generation infrastructure, combining NVFP4 training, KV-cache compression, parallel dequantization, and asynchronous VAE decoding. The training-side innovation is Balanced SP.

The Problem: AR Teacher Forcing Creates Imbalance

In autoregressive video generation, teacher forcing builds a training sequence by concatenating clean history latents and noisy target latents:

[ clean history latents ; noisy target latents ]

If standard Ulysses-style SP slices this concatenation without understanding the AR objective, some ranks get mostly clean context (no loss) while others get the noisy target tokens (all the loss). The sequence is partitioned but the training work is not:

Traditional SP:
  GPU 0: clean z0          ← no loss
  GPU 1: clean z1          ← no loss
  GPU 2: clean z2          ← no loss
  GPU 3: noisy z3 + loss   ← all the loss

This is a load imbalance that naively grows worse as the video gets longer.

The Solution: Paired Clean/Noisy Chunks on Each Rank

Balanced SP changes the work unit so each rank locally constructs both clean and noisy latents from the same temporal chunk:

Balanced SP:
  GPU 0: clean z0 + noisy z0 + local loss
  GPU 1: clean z1 + noisy z1 + local loss
  GPU 2: clean z2 + noisy z2 + local loss
  GPU 3: clean z3 + noisy z3 + local loss

Every rank now owns context tokens, target tokens, and a share of the loss. The teacher-forcing mask is constructed in Ulysses attention order from those clean/noisy identities without materializing a separate global permutation.

SP-Aware Chunked VAE Encoding

Critically, Balanced SP extends all the way back to the VAE. Instead of replicating the full VAE encoding on every rank:

  • Each rank VAE-encodes only its local raw-video chunk plus a small left halo covering the VAE temporal receptive field.
  • The halo latent is discarded after encoding; only the local chunk is kept.

This is a video-specific lesson: if the transformer is sharded but the VAE pipeline is replicated, the system has not actually made long-video training scale. SP must begin where the expensive sequence is created.

Performance Results

LongLive-2.0 reports NVFP4 + Balanced SP as the fastest training configuration:

Video Length Iteration Time Speedup vs BF16+SP
16s 40.1s 1.3×
32s 119.3s 1.4×
64s 639.5s 2.1×

Additional results: up to 2.15× training speedup, 1.84× inference speedup, and 45.7 FPS inference.

LongLive-2.0 takeaway: The right SP unit for long-video generation is a temporal chunk that owns clean history, noisy target, local VAE encoding with a small left overlap, Ulysses-order mask construction, and target tokens.

5. Understanding vs. Generation: Same Principle, Different Work Assignment

Design aspect LongVILA (Understanding) LongLive-2.0 (Generation)
Sequence unit Visual tokens + text tokens Video latent chunks
Main bottleneck Vision encoder workload, LLM context length Clean/noisy layout, target-token imbalance, VAE preparation
Why naive SP fails Text-only sharding ignores where visual tokens originate Concatenated clean/noisy sharding leaves loss on a few ranks
Core design MM-SP: shard by frames, then by tokens Balanced SP: each rank owns matched clean/noisy temporal chunks
General lesson Modality-aware work assignment Objective-aware temporal work assignment

The unifying abstraction is meaningful work assignment: a rank should own a slice that makes the upstream encoder, attention communication, loss computation, and hardware topology all behave well together.

6. Design Principles for Long-Video Training Systems

The blog distills five design principles that I find broadly applicable beyond video:

Principle 1: Shard the Real Bottleneck. Split the work that actually limits scale, not the easiest tensor. For LongVILA, that is frame/image encoding. For LongLive-2.0, that is VAE preparation and target-token distribution.

Principle 2: Keep the Training Objective Invariant. SP should not change temporal order, positions, attention visibility, loss masks, or which tokens are targets. If sharding alters what the model is trained to predict, the system is broken—no matter how well it scales.

Principle 3: Match the Hardware Topology. Ring, Ulysses, 2D-Attention, USP, and LoongTrain differ mainly in communication patterns. A good video system chooses All-to-All vs. P2P, and intra-node vs. inter-node traffic deliberately—not as an afterthought.

Principle 4: Start Before the Transformer. Video sequence construction begins before attention: frame loading, vision encoding, VAE encoding with overlap, latent chunking, mask construction. If SP starts only inside transformer blocks, the imbalance is already baked in.

Principle 5: Check What Each Rank Actually Handles. After sharding, every rank should have meaningful work. A quick decision tree:

Across samples?                          → DP / [FSDP](/fsdp/2026/03/02/vescale-fsdp.html#fsdp) / ZeRO
Inside one long sample?                  → SP / Context Parallelism
Mostly text, enough attention heads?     → Ulysses-style SP
Many nodes or beyond head limits?        → Ring / USP / 2D-Attention / LoongTrain
Heavy multimodal encoder work?           → MM-SP-style two-stage sharding
Clean/noisy AR video streams?            → Balanced-SP-style temporal work assignment

Insights

  1. “The temporal dimension is the new batch dimension.” This is the central thesis of the blog and I think it is exactly right. Once a single sample no longer fits on one GPU, the system must distribute work within the sample, and SP is how that distribution happens. As video models push to minutes and hours of footage, SP will be as essential as FSDP is today.

  2. Domain-Aware SP > Generic SP. The most important lesson from both LongVILA and LongLive-2.0 is that generic “split the sequence” SP is necessary but not sufficient. The SP partition must be semantically aware: aware of where tokens come from (modality-aware for understanding) and what they mean for the training objective (objective-aware for generation). This is a fundamentally different design constraint than text-only SP systems face.

  3. SP Must Start at the Data Source, Not at the Transformer. Both systems push the SP boundary upstream—LongVILA to the vision encoder, LongLive-2.0 to the VAE. If the most expensive parts of the pipeline (frame encoding, VAE encoding) are replicated across all ranks, the system has not truly parallelized the bottleneck. This principle likely generalizes to any multimodal training system.

  4. The Convergence of Parallelism Strategies. Looking at this alongside Megatron’s Context Parallelism, USP, and LoongTrain, the field is converging on the idea that the four parallelism axes (data, tensor, pipeline, sequence/context) are all necessary and must be composed carefully. The remaining open question is how to automatically choose the right composition for a given model, modality, and hardware topology—a problem that systems like Alpa and FlexFlow have explored for the first three axes but that sequence parallelism has not yet fully automated.