HunyuanVideo-1.5: An 8.3B-Parameter Video Diffusion Model Built for Consumer GPUs

How Tencent’s lightweight video generator packs training, inference, and fine-tuning into one open-source codebase

Training a high-quality video generation model typically means orchestrating hundreds of GPUs for days, and even inference often demands a cluster. That barrier keeps most engineers out of the field entirely. Tencent’s HunyuanVideo-1.5 repository attacks this directly: an 8.3B-parameter diffusion transformer that claims single-GPU inference on an RTX 4090, paired with a complete training and fine-tuning pipeline in one open-source codebase. The repository is mature and active, with roughly 120 commits and a steady stream of feature releases covering fp8 quantization, feature caching, and step distillation.

By the end of this article, you will understand how the architecture achieves this footprint, what the code actually does at each layer, and where the trade-offs bite. The codebase is substantial—a PyTorch project with a modular pipeline, a 3D causal VAE, a DiT transformer with sparse attention, and distributed training utilities—so we will focus on the decisions that matter: the sparse attention mechanism, the Muon optimizer, and the inference optimizations that make consumer-GPU generation feasible.

What HunyuanVideo-1.5 Is (and Isn’t)

HunyuanVideo-1.5 is an open-source video generation model with 8.3B parameters, developed by Tencent. It supports text-to-video and image-to-video generation at 480p and 720p resolutions, with an optional super-resolution pipeline that upscales output to 1080p. The repository ships both inference (generate.py) and training (train.py) entry points, plus a full pipeline class (HunyuanVideo_1_5_Pipeline) built on PyTorch and Hugging Face Diffusers.

The project is not a self-contained application. Model weights must be downloaded separately from Hugging Face, and optional features such as prompt rewriting require an external vLLM-compatible server. If that server is not configured, prompt rewriting silently skips. The codebase also assumes you will install optional CUDA kernels—FlashAttention, SageAttention, and flex-block-attn—depending on which attention mode you want.

Positioned at the model level, HunyuanVideo-1.5 provides the neural network architectures and orchestration pipelines. Above it sit applications like ComfyUI or your own scripts; below it are PyTorch, Diffusers, and the optional kernel libraries. The repository is research-oriented rather than a production service: it prioritizes flexibility and a wide range of optimization flags over turnkey simplicity. Expect to configure offloading, caching, and attention modes yourself, and to implement a custom dataloader if you intend to train.

Architecture: From Prompt to Video

The HunyuanVideo_1_5_Pipeline orchestrates the full generation flow. It loads the VAE, text encoders, transformer, vision encoder, and byT5 glyph processor, then coordinates encoding, denoising, and decoding. The pipeline auto-detects offloading based on GPU memory: below 60 GB it enables both model offloading and group offloading, while above that threshold it uses only model offloading.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

flowchart LR
    A[Text Prompt] --> B[LLM Text Encoder]
    A --> C[byT5 Glyph Encoder]
    D[Image Path] --> E[SigLIP Vision Encoder]
    B --> F[Latent Noise Init]
    C --> F
    E --> F
    F --> G[Denoising Loop<br/>Transformer + Scheduler]
    G --> H[3D Causal VAE Decode]
    H --> I[Optional Super-Resolution]
    I --> J[MP4 Output]

The HunyuanVideo_1_5_DiffusionTransformer is a DiT with double and single stream blocks. Double stream blocks process image and text tokens with separate attention branches before fusing, while single stream blocks concatenate them. The transformer supports multiple attention modes—flash, sageattn, and flex-block-attn—with automatic fallback to torch attention if an optional kernel is missing.

The 3D causal VAE compresses latents 16x spatially and 4x temporally. Its PatchCausalConv3d splits large tensors along the temporal axis when memory exceeds a threshold, and spatial tiling handles high resolutions. Temporal tiling is explicitly unsupported and raises a RuntimeError.

The FlowMatchDiscreteScheduler implements Euler steps with timestep shift, supporting both SD3-style and Flux-style shifts based on token count. The pipeline determines task type (t2v vs i2v) by the presence of an image path, and the denoising loop runs the transformer iteratively with optional feature caching and sparse attention.

For distributed training, parallel_states manages sequence parallelism and FSDP device meshes. The transformer uses all-to-all communication to redistribute attention heads across sequence-parallel groups, enabling training on longer sequences than a single GPU could hold.

Key Features: Optimizations That Make Consumer GPUs Viable

The core challenge for any video generation model is the sheer compute cost of attention over long spatiotemporal sequences. HunyuanVideo-1.5 addresses this with Selective and Sliding Tile Attention (SSTA), which prunes redundant key-value blocks rather than computing full attention. The implementation in hyvideo/models/transformers/modules/ssta_attention.py selects top-k blocks via importance or similarity sampling, reducing attention computation substantially for long videos. This comes with two constraints: it requires the flex-block-attn kernel, and it only runs on NVIDIA GPUs.

Feature caching provides a complementary speedup by exploiting redundancy across denoising timesteps. The CacheHelper in hyvideo/commons/cache_helper.py caches block outputs and skips forward passes when the current timestep and block ID indicate the output would be nearly identical. Three cache strategies are available—deepcache, teacache, and taylorcache—each trading a small quality loss for significant inference speedup.

Step distillation attacks the problem from the sampling side. A distilled model generates videos in 8 or 12 steps instead of the standard 50, cutting end-to-end generation time by up to 75% on an RTX 4090. Note that step distillation and feature caching are mutually exclusive; enabling both raises a ValueError.

For memory-constrained GPUs, fp8 GEMM inference quantizes transformer matrix multiplications to 8-bit floating point using sgl-kernel. This reduces memory footprint and accelerates computation on supported hardware. Model offloading and group offloading automatically move components to CPU based on available GPU memory, allowing the full pipeline to fit within 14GB VRAM.

On the training side, the Muon optimizer in hyvideo/optim/muon.py orthogonalizes gradient updates via Newton-Schulz iteration, accelerating convergence for large matrix parameters. It is paired with AdamW for 1D and embedding parameters, which do not benefit from orthogonalization. Finally, the byT5 glyph encoder improves text rendering quality for fonts and colors, though it adds a dependency on Glyph-SDXL-v2 and increases memory usage.

Use Cases: Where It Shines and Where It Doesn’t

The clearest fit is a developer with a single RTX 4090 who wants to generate videos from text prompts. The 8.3B-parameter model, combined with offloading, feature caching, and step distillation, fits within the 24GB VRAM of that card. The step-distilled variant runs in 8 or 12 steps instead of 50, cutting end-to-end generation time by up to 75% on the same hardware.

Researchers fine-tuning on custom video data are also well served. The repository ships train.py with FSDP, sequence parallelism, LoRA support, and the Muon optimizer. The training script includes configurable SNR sampling strategies and timestep shift, giving researchers control over the flow-matching objective without writing infrastructure from scratch.

Users who need legible text in their videos—specific fonts, colors, or multilingual glyphs—benefit from the byT5 glyph encoder. This is a differentiator over models that render text as noise; the encoder produces character-level embeddings that survive the denoising process.

The fit is only partial for production services with high throughput requirements. The repository is research-oriented: it lacks a specialized inference engine, requires manual configuration of optional kernels (FlashAttention, SageAttention, flex-block-attn), and expects users to tune offloading and caching flags themselves. You can get good throughput, but you will spend engineering time getting there.

Two scenarios are poor fits. Generating 1080p video directly is not supported; the main model produces 480p or 720p, and you must run the separate super-resolution pipeline afterward. That adds a second generation pass, more VRAM pressure, and longer wall-clock time. And if you do not have an NVIDIA GPU, sparse attention is unavailable—the code checks for 'nvidia h' in the device name—so you lose one of the key speedups that makes long-video generation practical on consumer hardware.

Interface and Usage: Real Commands from the Repo

Both inference and training are driven by two CLI entrypoints: generate.py and train.py. The inference script accepts a prompt, resolution, and model path, then handles task-type detection, offloading configuration, and optional optimizations internally.

1
2
3
4
5

python generate.py \
  --prompt 'A girl holding a paper with words "Hello, world!"' \
  --resolution 480p \
  --model_path ./ckpts \
  --output_path ./outputs/output.mp4

This runs the full text-to-video pipeline: text encoding, latent initialization, denoising, and VAE decode. For image-to-video, add --image_path /path/to/image.png; the script switches to i2v mode automatically when that argument is present.

Step distillation reduces inference steps dramatically. The distilled model runs in 8 steps instead of the default 50:

1
2
3
4
5
6

python generate.py \
  --prompt 'A cat playing with a ball' \
  --resolution 480p \
  --model_path ./ckpts \
  --enable_step_distill true \
  --num_inference_steps 8

Note that --enable_step_distill and --enable_cache are mutually exclusive; the script raises a ValueError if both are set.

Training is launched via train.py. The script supports full fine-tuning and LoRA, with FSDP enabled by default:

1
2
3
4
5
6

python train.py \
  --pretrained_model_root ./ckpts \
  --use_lora true \
  --lora_r 8 \
  --lora_alpha 16 \
  --output_dir ./lora_output

The training script ships with a dummy dataloader; you must replace create_dummy_dataloader() with your own implementation. The dataset’s __getitem__ must return pixel_values (video as [C, F, H, W] with F = 4n+1 frames, or image as [C, H, W]), a text prompt, and a data_type of "video" or "image".

For programmatic use, the pipeline class is available directly. create_pipeline() loads all components, then the returned object is called with generation parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

pipe = HunyuanVideo_1_5_Pipeline.create_pipeline(
    pretrained_model_name_or_path="./ckpts",
    transformer_version="480p_t2v",
    create_sr_pipeline=False,
    transformer_dtype=torch.bfloat16,
    device=torch.device("cuda"),
    transformer_init_device=torch.device("cuda"),
)
out = pipe(
    prompt="A cat playing with a ball",
    aspect_ratio="16:9",
    num_inference_steps=50,
    video_length=121,
    seed=1,
    output_type="pt",
)

Prompt rewriting is optional and requires a vLLM-compatible server. If no endpoint is configured, the pipeline silently skips rewriting rather than failing.

How It Compares: HunyuanVideo-1.5 vs. Alternatives

HunyuanVideo-1.5 sits in a crowded field of open-source video diffusion models, each with different trade-offs. The table below summarizes the key axes of comparison; where performance numbers are not published in the repository or its documentation, the entry reads “unknown.”

Compared to the original HunyuanVideo, the 1.5 release trades raw capacity for efficiency. The 8.3B-parameter model adds selective and sliding tile attention (SSTA), step distillation, and feature caching, all aimed at fitting consumer GPUs. The trade-off is lower quality potential than the 13B predecessor, though no benchmark numbers in the repository quantify the gap.

CogVideoX, under Apache 2.0, has a larger community and a permissive license. HunyuanVideo-1.5 counters with a more complete training pipeline—FSDP, sequence parallelism, and the Muon optimizer—and a stronger focus on consumer-GPU inference efficiency. For teams that need to fine-tune, the training support is a differentiator.

Wan2.1, also from Tencent, is the closest sibling. The two are not benchmarked against each other in the repository, and their architectural differences are not documented side by side. The comparison here reflects general knowledge and may be out of date; verify current capabilities before making a selection.

Under the Hood: SSTA and Muon in Detail

The two most distinctive mechanisms in HunyuanVideo-1.5 are Selective and Sliding Tile Attention (SSTA) for inference and the Muon optimizer for training. Both are implemented as modular components that can be swapped or extended without touching the surrounding pipeline.

SSTA replaces dense attention with a block-based sparse scheme. Instead of computing attention over every token pair, the implementation in hyvideo/models/transformers/modules/ssta_attention.py tiles the sequence into 3D blocks and selects only the top-k most relevant blocks per query. The importance_sampling function scores blocks by combining query-key similarity with a redundancy penalty:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

def importance_sampling(q, k, topk, threshold=0.0, lambda_=0.9, adaptive_pool=None):
    q = q / q.norm(dim=-1, keepdim=True)
    k = k / k.norm(dim=-1, keepdim=True)
    gate_similarity = torch.einsum("bhsd,bhkd->bhsk", q, k)
    gate_similarity = (gate_similarity + 1.0) / 2.0
    gate_unique = torch.einsum("bhsd,bhkd->bhsk", k, k)
    gate_unique = (gate_unique + 1.0) / 2.0
    
    B, H, K_num, D = k.shape
    diag_indices = torch.arange(K_num, device=k.device)
    gate_unique[:, :, diag_indices, diag_indices] = torch.nan
    
    mean_redundancy = torch.nanmean(gate_unique, dim=-2, keepdim=True)
    importance_scores = lambda_ * gate_similarity - (1 - lambda_) * mean_redundancy
    
    topk = min(topk, importance_scores.size(-1))
    _, top_block_indices = importance_scores.topk(k=topk, dim=-1, sorted=False)
    return top_block_indices

The similarity term rewards blocks that match the query; the redundancy term penalizes blocks that are similar to other key blocks, forcing the selection to cover diverse content. The lambda_ parameter (default 0.9) balances these two objectives. The function also supports text masking and adaptive pooling, letting the attention skip text tokens or pool features before scoring.

The selected block indices become a sparse mask passed to flex_block_attn_func, which computes attention only on the chosen blocks. This cuts KV computation substantially for long videos, where dense attention would otherwise scale quadratically with sequence length.

For training, the Muon optimizer in hyvideo/optim/muon.py takes a different approach to acceleration. For parameters with two or more dimensions, it orthogonalizes the gradient via Newton-Schulz iteration before applying the update:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def zeropower_via_newtonschulz5(G, steps = 5):
    assert len(G.shape) >= 2
    a, b, c = (3.4445, -4.7750, 2.0315)
    X = G
    if G.size(-2) > G.size(-1):
        X = X.mT
    X = X / (X.norm(dim=(-2, -1), keepdim=True) + 1e-7)
    for _ in range(steps):
        A = X @ X.T
        B = b * A + c * A @ A
        X = a * X + B @ X
    if G.size(-2) > G.size(-1):
        X = X.mT
    return X

The Newton-Schulz iteration computes an approximate orthogonalization of the gradient matrix, which the authors note produces updates with singular values in a bounded range rather than a true SVD.

What to take away

HunyuanVideo-1.5 demonstrates that high-quality video generation does not require a 13B+ parameter model. The 8.3B DiT, combined with SSTA sparse attention and feature caching, fits on consumer GPUs with 14GB VRAM while maintaining competitive output quality. The project’s real contribution is showing how to make large diffusion models practical on limited hardware: prune redundant attention computation, cache block outputs across timesteps, and distill inference steps aggressively.

The training pipeline is equally instructive. Muon with Newton-Schulz orthogonalization accelerates convergence for large matrix parameters, and the modular design—sequence parallelism, FSDP, and a pluggable dataloader—makes fine-tuning approachable. The codebase is cleanly separated into pipeline, model, scheduler, and distributed layers, which makes it straightforward to swap in alternative attention kernels or caching strategies.

Be honest about the gaps. The repository provides no quantitative benchmarks—no FPS numbers, no memory measurements, no quality comparisons against CogVideoX or Wan2.1. The training script ships with a dummy dataloader, so real training requires implementing your own data pipeline. Sparse attention depends on flex-block-attn and NVIDIA GPUs, and the custom Tencent license may restrict commercial use.

The code is worth studying regardless. Clone the repository at https://github.com/Tencent-Hunyuan/HunyuanVideo-1.5 and read ssta_attention.py and muon.py first—those two files contain the most transferable ideas.

What this analysis could not determine

• Exact performance benchmarks (e.g., FPS, memory usage) are not provided in the digest.
• The quality of the generated videos compared to other models is not quantitatively assessed in the digest.
• The training script’s dummy dataloader means the actual training data format is not fully demonstrated.
• The specific details of the SSTA attention mask generation (e.g., how topk is chosen) are partially visible but not fully explained.
• The license terms are custom and not detailed in the digest.

Further diagrams