EdenLive

EdenLive: Real-Time Generative Rendering from Any Visual Reference

-orange)

Project Page · Code · Models

Zehuan Huang1,, Tianxing Xu1,, Chunli Peng1,, Cheng Zhang1,, Yifei Song1,†, Zeyi Zheng1,†, Zekai Gu1,†, Zhipeng Yu1, Peng Wang1,‡, Yanpei Cao1,✉

* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author

Overview

EdenLive renders a proxy or reference video stream, a target first frame, and a text prompt into refined or stylized video. References can come from 2D/3D scenes, existing engines, or recordings. Upstream code maintains interaction state; the model handles visual rendering.

Control is injected through a vision encoder: a post-trained DINOv3 encoder embeds each proxy frame, and those features condition the diffusion transformer.

The project page demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P.

This repository holds the model weights. Code, installation instructions, and inference/streaming usage live in the EdenLive repository.

Model weights

hf download VAST-AI/EdenLive --local-dir ./models/EdenLive

Everything loads from one directory, models/EdenLive under the code repository root by default (about 210 GB in total); nothing is pulled from the Hub at run time once downloaded.

models/EdenLive/
  dit-14b/            diffusion_pytorch_model.safetensors, config.json  14B few-step (chunk-AR) DiT
  dit-14b-bidir/      diffusion_pytorch_model.safetensors, config.json  14B bidirectional DiT
  dit-5b/             diffusion_pytorch_model.safetensors, config.json  5B few-step (chunk-AR) DiT
  dit-5b-bidir/       diffusion_pytorch_model.safetensors, config.json  5B bidirectional DiT
  vae-14b/            Wan2.1_VAE.pth, config.json
  vae-5b/             Wan2.2_VAE.pth, config.json
  decoder-14b/        lighttaew2_1.pth                    TAEHV decoder (real-time only)
  decoder-5b/         lighttaew2_2.pth                    TAEHV decoder (real-time only)
  scheduler-14b/      scheduler_config.json               offline only
  scheduler-5b/       scheduler_config.json               offline only
  umt5/               text_encoder/, tokenizer/           UMT5 text encoder
  clip/               image_encoder/, image_processor/    CLIP, 14B I2V only
  vision-encoder/     backbone/, projection.safetensors   proxy-DINO encoder
  flux-c2f/           Diffusers FLUX tree + c2f_lora.safetensors   workbench first frame (real-time only)
  model_index.json

To load the weights from another directory:

python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive

The real-time service reads the same tree through MODEL_ROOT in realtime/deploy.conf. See the main repository's README for how each component is wired into the offline and real-time pipelines.

Usage

Each DiT has a matching inference profile in the code repository:

Few-step inference (chunk-autoregressive sampling, 4 steps per chunk) on the bundled samples:

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_few_step.yaml \
  runtime.sequence_parallel.size=2

python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml

See the Quick start for bidirectional inference, the real-time service, and your own data.

Citation

@misc{edenlive2026,
  title  = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
  author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Yifei Song and Zeyi Zheng and Zhipeng Yu and Peng Wang and Yanpei Cao},
  year   = {2026},
  url    = {https://vast-eden.ai/?p=edenlive}
}

Acknowledgements

Built on Wan2.1 / Wan2.2, 🤗 Diffusers, DINOv3, FLUX.2 [klein] by Black Forest Labs, the LightX2V LightTAE decoders, and LAION CLIP ViT-H/14 trained with OpenCLIP.

License

The weights in this repository are covered by different licenses, per component: