HunyuanVideo-I2V

中文阅读

HunyuanVideo-I2V 🌅

Following the great successful open-sourcing of our HunyuanVideo, we proudly present the HunyuanVideo-I2V, a new image-to-video generation framework to accelerate open-source community exploration!

This repo contains offical PyTorch model definitions, pre-trained weights and inference/sampling code. You can find more visualizations on our project page. Meanwhile, we have released the LoRA training code for customizable special effects, which can be used to create more interesting video effects.

HunyuanVideo: A Systematic Framework For Large Video Generation Model

🔥🔥🔥 News!!

• Mar 13, 2025: 🚀 We release the parallel inference code for HunyuanVideo-I2V powered by xDiT.
• Mar 07, 2025: 🔥 We have fixed the bug in our open-source version that caused ID changes. Please try the new model weights of HunyuanVideo-I2V to ensure full visual consistency in the first frame and produce higher quality videos.
• Mar 06, 2025: 👋 We release the inference code and model weights of HunyuanVideo-I2V. Download.

📑 Open-source Plan

• HunyuanVideo-I2V (Image-to-Video Model)
  • [x] Inference
  • [x] Checkpoints
  • [x] ComfyUI
  • [x] Lora training scripts
  • [x] Multi-gpus Sequence Parallel inference (Faster inference speed on more gpus)
  • [ ] Diffusers

Contents

• HunyuanVideo-I2V 🌅
  • 🔥🔥🔥 News!!
  • 📑 Open-source Plan
  • Contents
  • HunyuanVideo-I2V Overall Architecture
  • 📜 Requirements
  • 🛠️ Dependencies and Installation
    • Installation Guide for Linux
  • 🧱 Download Pretrained Models
  • 🔑 Single-gpu Inference
    • Tips for Using Image-to-Video Models
    • Using Command Line
    • More Configurations
  • 🎉 Customizable I2V LoRA effects training
    • Requirements
    • Environment
    • Training data construction
    • Training
    • Inference
  • 🚀 Parallel Inference on Multiple GPUs by xDiT
    • Using Command Line
  • 🔗 BibTeX
  • Acknowledgements

HunyuanVideo-I2V Overall Architecture

Leveraging the advanced video generation capabilities of HunyuanVideo, we have extended its application to image-to-video generation tasks. To achieve this, we employ a token replace technique to effectively reconstruct and incorporate reference image information into the video generation process.

Since we utilizes a pre-trained Multimodal Large Language Model (MLLM) with a Decoder-Only architecture as the text encoder, we can significantly enhance the model's ability to comprehend the semantic content of the input image and to seamlessly integrate information from both the image and its associated caption. Specifically, the input image is processed by the MLLM to generate semantic image tokens. These tokens are then concatenated with the video latent tokens, enabling comprehensive full-attention computation across the combined data.

The overall architecture of our system is designed to maximize the synergy between image and text modalities, ensuring a robust and coherent generation of video content from static images. This integration not only improves the fidelity of the generated videos but also enhances the model's ability to interpret and utilize complex multimodal inputs. The overall architecture is as follows.

📜 Requirements

The following table shows the requirements for running HunyuanVideo-I2V model (batch size = 1) to generate videos:

• An NVIDIA GPU with CUDA support is required.
  • The model is tested on a single 80G GPU.
  • Minimum: The minimum GPU memory required is 60GB for 720p.
  • Recommended: We recommend using a GPU with 80GB of memory for better generation quality.
• Tested operating system: Linux

🛠️ Dependencies and Installation

Begin by cloning the repository:

git clone https://github.com/tencent/HunyuanVideo-I2V
cd HunyuanVideo-I2V

Installation Guide for Linux

We recommend CUDA versions 12.4 or 11.8 for the manual installation.

Conda's installation instructions are available here.

# 1. Create conda environment
conda create -n HunyuanVideo-I2V python==3.11.9

# 2. Activate the environment
conda activate HunyuanVideo-I2V

# 3. Install PyTorch and other dependencies using conda
# For CUDA 12.4
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.4 -c pytorch -c nvidia

# 4. Install pip dependencies
python -m pip install -r requirements.txt

# 5. Install flash attention v2 for acceleration (requires CUDA 11.8 or above)
python -m pip install ninja
python -m pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.6.3

# 6. Install xDiT for parallel inference (It is recommended to use torch 2.4.0 and flash-attn 2.6.3)
python -m pip install xfuser==0.4.0

In case of running into float point exception(core dump) on the specific GPU type, you may try th