Open-Sora: A Config-Driven Video Diffusion Training and Inference Codebase

How the registry pattern, ColossalAI integration, and data pipeline fit together to make text-to-video research reproducible.

Training a video diffusion model from scratch means assembling a dataset pipeline, a scalable architecture, and a distributed training loop that all agree on the same tensor shapes and conditioning signals. Each piece is well-documented in isolation, but integrating them into a reproducible system is where most projects stall. Open-Sora (github.com/hpcaitech/Open-Sora) is an open-source codebase that packages this entire stack into a config-driven PyTorch framework. It is actively maintained, with versioned releases from v1.0 through v1.3 and recent commits as of February 2025.

The codebase is substantial—roughly 29,000 GitHub stars and a multi-directory layout spanning model definitions, schedulers, data tools, and entry-point scripts. Its design centers on a registry pattern that lets you swap models, schedulers, and datasets purely through Python config files. By the end of this deep dive, you will understand how that registry works, how ColossalAI provides sequence parallelism and ZeRO for training, and how the data pipeline feeds processed video-text pairs into the training loop.

What Open-Sora Is (and Isn’t)

Open-Sora is a training and inference codebase for text-to-video diffusion models, not a hosted service or a single pretrained model. It provides the full stack needed to build and run video generation systems: model architectures (STDiT, PixArt, DiT, Latte), VAE variants, diffusion schedulers, data processing tools, and a Gradio demo. Users supply their own data, GPU resources, and training runs.

The codebase sits above PyTorch and ColossalAI. PyTorch provides the tensor computation layer; ColossalAI handles distributed training with ZeRO and sequence parallelism. Open-Sora integrates with Hugging Face Transformers for text encoding (T5, CLIP) and uses components from the diffusers library for VAE functionality. Configuration is driven by mmengine Python config files, which specify model architecture, scheduler, dataset, and hyperparameters without code changes.

The project is actively developed, with versioned releases from v1.0 through v1.3 and recent commits as of February 2025. A distinguishing feature is its complete data pipeline: tools for scene cutting, captioning, scoring, and filtering video datasets are included in the repository. This is rare among open-source video generation projects, which typically focus on inference only.

This contrasts with closed-source alternatives like Stable Video Diffusion, which do not provide training code. Open-Sora’s value proposition is that researchers and engineers can train custom video models on their own data rather than being limited to a fixed pretrained model or proprietary API.

Architecture: A Registry-Driven Modular System

Open-Sora organizes its components into four layers: entrypoints, control plane, model & runtime, and data plane. Entrypoints are user-facing scripts (scripts/inference.py, scripts/train.py, gradio/app.py) that parse configs and orchestrate execution. The control plane handles module construction, distributed setup, and configuration validation. The model & runtime layer contains neural network definitions and diffusion sampling logic. The data plane provides video loading, transformation, and dataset preparation utilities.

The registry pattern in opensora/registry.py is the architectural linchpin. Built on mmengine’s Registry, it maintains central registries for models (MODELS), schedulers (SCHEDULERS), and datasets (DATASETS). The build_module function instantiates any component from a config dict, decoupling string type names from concrete classes. This allows config files to swap components without code changes—a researcher can substitute STDiT3 for PixArt or DPM-Solver for IDDPM by editing a single config field.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

flowchart TD
    subgraph Entrypoints
        CLI["scripts/inference.py"]
        TRAIN["scripts/train.py"]
        GRADIO["gradio/app.py"]
    end

    subgraph Control Plane
        REG["opensora/registry.py<br/>build_module, MODELS, SCHEDULERS"]
        CFG["opensora/utils/config_utils.py<br/>config validation"]
        ACCEL["opensora/acceleration/<br/>distributed init, checkpointing"]
    end

    subgraph Model & Runtime
        STDiT["STDiT3<br/>opensora/models/stdit/stdit3.py"]
        VAE["VAE v1.3<br/>opensora/models/vae_v1_3/"]
        SCHED["Schedulers<br/>IDDPM, DPM-Solver, Rectified Flow"]
        T5["T5/CLIP encoders"]
    end

    subgraph Data Plane
        DATA["opensora/datasets/"]
        TOOLS["tools/<br/>scene cut, captioning, scoring"]
    end

    CLI -->|"build_module"| REG
    TRAIN -->|"build_module"| REG
    GRADIO -->|"build_module"| REG
    REG --> STDiT
    REG --> VAE
    REG --> SCHED
    REG --> T5
    CLI -->|"scheduler.sample"| SCHED
    SCHED -->|"model.forward"| STDiT
    TRAIN -->|"booster.boost"| ACCEL
    STDiT -->|"auto_grad_checkpoint"| ACCEL
    CLI --> DATA
    TRAIN --> DATA
    TOOLS --> DATA
    VAE -->|"encode/decode"| CLI

The inference data flow illustrates how these layers interact. scripts/inference.py loads a mmengine Config, then calls build_module to construct the VAE, text encoder, and scheduler from the MODELS and SCHEDULERS registries. The STDiT3 diffusion model is built with latent size and text encoder dimensions, then loaded with pretrained weights.

Key Features: Solving Real Problems

Text-to-video generation is the core capability. The diffusion transformer (STDiT3) denoises a latent tensor conditioned on text embeddings from T5 or CLIP, with the sampling loop driven by a configurable scheduler—IDDPM, DPM-Solver, or Rectified Flow. This removes the need to implement a diffusion backbone from scratch; swapping schedulers is a config change, not a code change.

Image-to-video and video-to-video use mask-based conditioning. The mask_index parameter specifies which frames are reference frames (kept fixed) and which are generated. For i2v, a single reference image is encoded to latent space and placed at frame zero; for v2v, a set of reference frames anchors the output. This lets you animate a still or edit an existing clip without retraining or architectural changes.

Multi-resolution and aspect ratio support covers 144p through 4k. The prepare_multi_resolution_info function generates conditioning tensors for resolution, aspect ratio, and fps, which the model consumes as additional inputs. This matters for production: a 16:9 landscape video for YouTube and a 9:16 portrait for Shorts are generated from the same checkpoint, not separate models.

The data processing pipeline under tools/ handles scene cutting, captioning, scoring, and filtering. These scripts transform raw video collections into cleaned, captioned training pairs. Without this, you would build your own preprocessing stack—a substantial project in itself. The pipeline is the difference between a usable dataset and a pile of unlabeled footage.

Distributed training with ColossalAI provides ZeRO, sequence parallelism, and gradient checkpointing. Sequence parallelism splits the attention sequence dimension across GPUs, which is essential for long videos where a single GPU cannot hold the activations. Gradient checkpointing trades compute for memory, letting you fit larger batch sizes. These features remove the memory ceiling that typically forces researchers to shorten videos or shrink models.

The Gradio demo in gradio/app.py exposes the full generation pipeline through a web UI. Users select resolution, aspect ratio, length, and mode (t2v/i2v/v2v) without touching config files or the command line. It lowers the barrier for quick experimentation and for demonstrating results to collaborators who are not engineers.

Use Cases: Where It Fits and Where It Doesn’t

Open-Sora is a strong fit for researchers who want to train a custom video generation model on their own dataset. The repository provides a complete pipeline: data preprocessing tools for scene cutting, captioning, and filtering; configurable model architectures registered through a central registry; and distributed training via ColossalAI with ZeRO and sequence parallelism. A researcher can go from raw video files to a trained model without assembling components from multiple projects.

Developers who simply want to generate videos from text prompts using a pretrained model will also find the codebase accessible. The inference script accepts a config file and produces an mp4, and the Gradio demo provides an interactive interface for testing prompts without writing code. For this use case, the learning curve is limited to understanding which config fields control resolution, length, and sampling parameters.

The fit is partial for a company deploying a service with specific aspect ratios. Multi-resolution support and aspect ratio buckets exist, but generating at high resolutions requires substantial GPU memory. The codebase assumes multi-GPU setups for anything beyond short, low-resolution clips, so a production deployment would need to budget for that hardware.

Open-Sora is a poor fit for a hobbyist with a single GPU. Training is heavy even at reduced scale, inference at higher resolutions will be slow, and the setup involves multiple dependencies (ColossalAI, mmengine, flash-attn) that are non-trivial to configure. The Gradio demo is the most accessible entry point, but it hardcodes local paths for model weights—for example, /home/guoxinying/...—which must be edited before external use. Additionally, the T5 text encoder must remain in fp32 for numerical stability; casting it to bf16 or fp16 will cause sampling to fail.

Interface and Usage: Config-Driven Scripts

The primary entry points are three command-line scripts. Inference runs with python scripts/inference.py --config <config>, training with python scripts/train.py --config <config>, and the interactive demo with python gradio/app.py --model-type v1.3. All three are thin wrappers around mmengine Python config files, which specify the model architecture, VAE, scheduler, dataset, and hyperparameters.

Configuration uses mmengine’s Python-based Config class rather than YAML or JSON. This is flexible—configs can compute values, import modules, and compose dicts—but it means config files are arbitrary Python code and execute at load time. The parse_configs function in opensora/utils/config_utils.py validates required fields and dtype choices before anything else runs.

The core construction mechanism is build_module from opensora/registry.py. It takes a config dict and a registry, then instantiates the corresponding class. In scripts/inference.py, each component is built in sequence:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

text_encoder = build_module(cfg.text_encoder, MODELS, device=device)
vae = build_module(cfg.vae, MODELS).to(device, dtype).eval()
model = (
    build_module(
        cfg.model,
        MODELS,
        input_size=latent_size,
        in_channels=vae.out_channels,
        caption_channels=text_encoder.output_dim,
        model_max_length=text_encoder.model_max_length,
        enable_sequence_parallelism=enable_sequence_parallelism,
    )
    .to(device, dtype)
    .eval()
)
scheduler = build_module(cfg.scheduler, SCHEDULERS)

The registry pattern decouples config string names from concrete classes. To swap the diffusion model from STDiT to PixArt, you change the model key in the config; no code changes are needed.

After building components, inference prepares prompts, computes latent size from the VAE, and enters the sampling loop. The key API call is scheduler.sample, which runs the full denoising loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

samples = scheduler.sample(
    model,
    text_encoder,
    z=z,
    z_cond=ref,
    z_cond_mask=x_cond_mask,
    prompts=batch_prompts_loop,
    device=device,
    additional_args=model_args,
    progress=verbose >= 2,
    mask=masks,
    mask_index=mask_index,
    image_cfg_scale=image_cfg_scale,
    neg_prompts=neg_prompts_batch_cl if mask_index is None else None,
    use_sdedit=use_sdedit,
    use_oscillation_guidance_for_text=use_oscillation_guidance_for_text,
    use_oscillation_guidance_for_image=use_oscillation_guidance_for_image,
)

The z argument is the initial random latent; z_cond and mask_index enable image-to-video and video-to-video conditioning. The returned samples are latent tensors, which are then decoded by vae.decode and saved as mp4 files.

One gotcha: setup.py declares version 1.2.0 even though the latest release is v1.3. Check the git tags or release notes rather than the package version when verifying which model family you are running.

How It Compares: Open-Sora vs. Alternatives

The following comparison reflects general knowledge of these projects as of early 2025 and may be out of date. Where the analysis does not provide specific data, the value is marked “unknown.”

The most significant differentiator is openness. Stable Video Diffusion is a closed model with no training code, which makes it unsuitable for researchers who need to train or fine-tune on custom data. VideoCrafter is open-source but provides only a partial training pipeline. Open-Sora ships a complete pipeline from data preprocessing through distributed training to inference.

Architecturally, Open-Sora uses a transformer-based STDiT model, whereas both alternatives rely on UNet backbones. This is not merely an implementation detail: transformer architectures scale differently with sequence length and integrate more naturally with sequence parallelism.

Open-Sora’s distributed training via ColossalAI—with ZeRO and sequence parallelism—is a practical differentiator for teams with multi-GPU resources. Exact performance numbers relative to the alternatives are unknown; no benchmark data is documented in the codebase.

Notable Techniques: Sequence Parallelism and Tiled Convolutions

Open-Sora scales to long videos through two complementary memory-reduction techniques. Sequence parallelism splits the sequence dimension of attention across GPUs, while tiled convolutions partition spatial-temporal inputs during VAE encoding and decoding.

Sequence parallelism is implemented in opensora/acceleration/communications.py and consumed by the STDiT3 model in opensora/models/stdit/stdit3.py. The core function, split_forward_gather_backward, distributes the sequence across devices during the forward pass and gathers results during backpropagation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

def split_forward_gather_backward(tensor, seq_parallel_group=None, grad_scale=1.0):
    if seq_parallel_group is None:
        return tensor
    group_size = dist.get_world_size(seq_parallel_group)
    if group_size == 1:
        return tensor
    rank = dist.get_rank(seq_parallel_group)
    seq_len = tensor.shape[1]
    split_size = seq_len // group_size
    tensor_list = [
        tensor[:, rank * split_size : (rank + 1) * split_size].contiguous()
    ]
    dist.all_gather(tensor_list, tensor, group=seq_parallel_group)
    tensor = torch.cat(tensor_list, dim=1)
    return tensor

The function splits the sequence dimension into contiguous chunks per rank, then uses all_gather to reconstruct the full sequence. This keeps each GPU’s activation memory proportional to seq_len / world_size during attention computation, at the cost of an all-gather communication step.

For the VAE, opensora/models/layers/tiled_conv3d.py provides a TiledConv3d class that partitions input along the largest spatial or temporal dimension. It computes convolution output positions per tile, applies padding ahead of time, and concatenates partial results. The class also includes from_native_conv3d, a static factory that converts a standard nn.Conv3d to its tiled variant while preserving weights.

The VAE additionally uses causal 3D convolutions with asymmetric padding (CausalConv3dPlainAR in opensora/models/vae_v1_3/modules/conv.py), which pad only past frames to enable autoregressive-style encoding and decoding of video sequences. For sampling, Open-Sora offers rectified flow and DPM-Solver schedulers as fast ODE-based alternatives to standard DDPM, reducing the number of inference steps required.

What to take away

Open-Sora demonstrates that a config-driven, registry-based design scales across the full video-generation stack—from data preprocessing to distributed training to interactive inference. The build_module pattern in opensora/registry.py lets you swap models, schedulers, and datasets by editing a Python config file rather than touching code. That separation of configuration from implementation is the project’s most transferable architectural lesson.

The techniques for scaling video models to long sequences are also worth studying. Sequence parallelism splits attention across GPUs, and tiled 3D convolutions partition VAE encoding along a chosen dimension to bound peak memory. Both are practical answers to the core problem of video diffusion: tensors grow quickly along the temporal axis.

Be honest about the gaps. The repository does not document training time or inference speed on specific hardware, and exact parameter counts for v1.3 are not stated. The Gradio app hardcodes local weight paths that must be edited for external use, and the data pipeline pulls in many heavy dependencies. If you want production-grade output quality comparable to closed models, you will need to invest in data curation and compute yourself.

The codebase is actively maintained, with releases from v1.0 through v1.3 and recent commits as of early 2025. The repository is at github.com/hpcaitech/Open-Sora; the README and configs/ directory are the best starting points for understanding the current state of the project.

What this analysis could not determine

• Exact model weight sizes and parameter counts for v1.3 are not in the digest.
• The performance (training time, inference speed) on specific hardware is not documented in the provided files.
• The precise differences between v1.2 and v1.3 model architectures beyond the VAE and STDiT3 are not fully detailed.
• The data processing pipeline’s exact behavior for all CLI flags is not fully described in the digest.

Further diagrams