lightx2v/Autoencoders

🎨 LightVAE

⚡ Efficient Video Autoencoder (VAE) Model Collection

From Official Models to Lightx2v Distilled Optimized Versions - Balancing Quality, Speed and Memory

  

For VAE, the LightX2V team has conducted a series of deep optimizations, deriving two major series: LightVAE and LightTAE, which significantly reduce memory consumption and improve inference speed while maintaining high quality.

💡 Core Advantages

<table> <tr> <td width="50%">

📊 Official VAE

Features: Highest Quality ⭐⭐⭐⭐⭐

✅ Best reconstruction accuracy ✅ Complete detail preservation ❌ Large memory usage (~8-12 GB) ❌ Slow inference speed

</td> <td width="50%">

🚀 Open Source TAE Series

Features: Fastest Speed ⚡⚡⚡⚡⚡

✅ Minimal memory usage (~0.4 GB) ✅ Extremely fast inference ❌ Average quality ⭐⭐⭐ ❌ Potential detail loss

</td> </tr> <tr> <td width="50%">

🎯 LightVAE Series (Our Optimization)

Features: Best Balanced Solution ⚖️

✅ Uses Causal 3D Conv (same as official) ✅ Quality close to official ⭐⭐⭐⭐ ✅ Memory reduced by ~50% (~4-5 GB) ✅ Speed increased by 2-3x ✅ Balances quality, speed, and memory 🏆

</td> <td width="50%">

⚡ LightTAE Series (Our Optimization)

Features: Fast Speed + Good Quality 🏆

✅ Minimal memory usage (~0.4 GB) ✅ Extremely fast inference ✅ Quality close to official ⭐⭐⭐⭐ ✅ Significantly surpasses open source TAE

</td> </tr> </table>

📦 Available Models

🎯 Wan2.1 Series VAE

🎯 Wan2.2 Series VAE

📊 Wan2.1 Series Performance Comparison

• —Precision: BF16
• —Test Hardware: NVIDIA H100

Video Reconstruction (5s 81-frame video)

Video Generation

Task: s2v(speech to video) Model: seko-talk

<table> <tr> <td width="25%" align="center"> <strong>Wan2.1VAE</strong><br> <video controls autoplay muted width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/6l-P-3Hr9JKL3xgUyJXWJ.mp4"></video> </td> <td width="25%" align="center"> <strong>taew21</strong><br> <video controls autoplay muted width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/rcVHrCKB4nRAs2VSjJd2d.mp4"></video> </td> <td width="25%" align="center"> <strong>lighttaew21</strong><br> <video controls autoplay muted width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/Wq9p9Z7NDYwaKw4SqVbYT.mp4"></video> </td> <td width="25%" align="center"> <strong>lightvaew21</strong><br> <video controls autoplay muted width="100%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/NpKOzFcvsHzSFfFACzUKP.mp4"></video> </td> </tr> </table>

📊 Wan2.2 Series Performance Comparison

• —Precision: BF16
• —Test Hardware: NVIDIA H100

Video Reconstruction

Video Generation

Task: t2v(text to video) Model: Wan2.2-TI2V-5B

<table> <tr> <td width="33%" align="center"> <strong>Wan2.2VAE</strong><br> <video controls autoplay width="95%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/KUY7Ifz9gFJqDjWga6A53.mp4"></video> </td> <td width="33%" align="center"> <strong>taew22</strong><br> <video controls autoplay width="95%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/OYA8VfNlCvhBkjnOMl.mp4"></video> </td> <td width="33%" align="center"> <strong>lighttaew22</strong><br> <video controls autoplay width="95%" src="https://cdn-uploads.huggingface.co/production/uploads/680de13385293771bc57400b/gaHRr6uuAF0NlH4YlMbHO.mp4"></video> </td> </tr> </table>

🎯 Model Selection Recommendations

Selection by Use Case

<table> <tr> <td width="33%">

🏆 Pursuing Best Quality

Recommended: Wan2.1_VAE / Wan2.2_VAE

• —✅ Official model, quality ceiling
• —✅ Highest reconstruction accuracy
• —✅ Suitable for final product output
• —⚠️ Large memory usage (~8-12 GB)
• —⚠️ Slow inference speed

</td> <td width="33%">

⚖️ Best Balance 🏆

Recommended: `lightvaew2_1`

• —✅ Uses Causal 3D Conv (same as official)
• —✅ Excellent quality, close to official
• —✅ Memory reduced by ~50% (~4-5 GB)
• —✅ Speed increased by 2-3x
• —✅ Close to official quality ⭐⭐⭐⭐

Use Cases: Daily production, strongly recommended ⭐

</td> <td width="33%">

⚡ Speed + Quality Balance ✨

Recommended: `lighttaew2_1` / `lighttaew2_2`

• —✅ Extremely low memory usage (~0.4 GB)
• —✅ Extremely fast inference
• —✅ Quality significantly surpasses open source TAE
• —✅ Close to official quality ⭐⭐⭐⭐

Use Cases: Development testing, rapid iteration

</td> </tr> </table>

🔥 Our Optimization Results Comparison

📑 Todo List

• —[x] LightX2V integration
• —[x] ComfyUI integration
• —[ ] Training & Distillation Code

🚀 Usage

Download VAE Models

# Download Wan2.1 official VAE
huggingface-cli download lightx2v/Autoencoders \
    --local-dir ./models/vae/

🧪 Video Reconstruction Test

We provide a standalone script vid_recon.py to test VAE models independently. This script reads a video, encodes it through VAE, then decodes it back to verify the reconstruction quality.

Script Location: LightX2V/lightx2v/models/video_encoders/hf/vid_recon.py

git clone https://github.com/ModelTC/LightX2V.git
cd LightX2V

1. Test Official VAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/Wan2.1_VAE.pth \
    --model_type vaew2_1 \
    --device cuda \
    --dtype bfloat16

2. Test Official VAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/Wan2.2_VAE.pth \
    --model_type vaew2_2 \
    --device cuda \
    --dtype bfloat16

3. Test LightTAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lighttaew2_1.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

4. Test LightTAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lighttaew2_2.pth \
    --model_type taew2_2 \
    --device cuda \
    --dtype bfloat16

5. Test LightVAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/lightvaew2_1.pth \
    --model_type vaew2_1 \
    --device cuda \
    --dtype bfloat16 \
    --use_lightvae

6. Test TAE (Wan2.1)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/taew2_1.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

7. Test TAE (Wan2.2)

python -m lightx2v.models.video_encoders.hf.vid_recon \
    input_video.mp4 \
    --checkpoint ./models/vae/taew2_2.pth \
    --model_type taew2_1 \
    --device cuda \
    --dtype bfloat16

Use in LightX2V

Specify the VAE path in the configuration file:

Using Official VAE Series:

{

    "vae_path": "./models/vae/Wan2.1_VAE.pth"
}

Using LightVAE Series:

{
    "use_lightvae": true,
    "vae_path": "./models/vae/lightvaew2_1.pth"
}

Using LightTAE Series:

{
    "use_tae": true,
    "need_scaled": true,
    "tae_path": "./models/vae/lighttaew2_1.pth"
}

Using TAE Series:

{
    "use_tae": true,
    "tae_path": "./models/vae/taew2_1.pth"
}

Then run the inference script:

cd LightX2V/scripts
bash wan/run_wan_i2v.sh  # or other inference scripts

Use in ComfyUI

please refer to https://github.com/ModelTC/ComfyUI-LightVAE

⚠️ Important Notes

1. Compatibility

• —Wan2.1 series VAE only works with Wan2.1 backbone models
• —Wan2.2 series VAE only works with Wan2.2 backbone models
• —Do not mix different versions of VAE and backbone models

📚 Related Resources

Documentation Links

• —LightX2V Quick Start: Quick Start Documentation
• —Model Structure Description: Model Structure Documentation
• —taeHV Project: GitHub - madebyollin/taeHV

Related Models

• —Wan2.1 Backbone Models: Wan-AI Model Collection
• —Wan2.2 Backbone Models: Wan-AI/Wan2.2-TI2V-5B
• —LightX2V Optimized Models: lightx2v Model Collection

🤝 Community & Support

• —GitHub Issues: https://github.com/ModelTC/LightX2V/issues
• —HuggingFace: https://huggingface.co/lightx2v
• —LightX2V Homepage: https://github.com/ModelTC/LightX2V

If you find this project helpful, please give us a ⭐ on GitHub