hw-liang/Diffusion4D

Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models

[[Project Page]](https://vita-group.github.io/Diffusion4D/) | [[Arxiv]](https://arxiv.org/abs/2405.16645) | [[Code]](https://github.com/VITA-Group/Diffusion4D)

News

• —2024.6.28: Released rendered data from curated objaverse-xl.
• —2024.6.4: Released rendered data from curated objaverse-1.0, including orbital videos of dynamic 3D, orbital videos of static 3D, and monocular videos from front view.
• —2024.5.27: Released metadata for objects!

Overview

We collect a large-scale, high-quality dynamic 3D(4D) dataset sourced from the vast 3D data corpus of Objaverse-1.0 and Objaverse-XL. We apply a series of empirical rules to filter the dataset. You can find more details in our paper. In this part, we will release the selected 4D assets, including:

1. 1.Selected high-quality 4D object ID.
2. 2.A render script using Blender, providing optional settings to render your personalized data.
3. 3.Rendered 4D images by our team to save your GPU time. With 8 GPUs and a total of 16 threads, it took 5.5 days to render the curated objaverse-1.0 dataset.

4D Dataset ID/Metadata

We collect 365k dynamic 3D assets from Objaverse-1.0 (42k) and Objaverse-xl (323k). Then we curate a high-quality subset to train our models.

Metadata of animated objects (323k) from objaverse-xl can be found in meta_xl_animation_tot.csv. We also release the metadata of all successfully rendered objects from objaverse-xl's Github subset in meta_xl_tot.csv. For customized blender from objaverse-xl, please use the curated xl data list objaverseXL_curated_uuid_list.txt.

For text-to-4D generation, the captions are obtained from the work Cap3D.

Citation

If you find this repository/work/dataset helpful in your research, please consider citing the paper and starring the repo ⭐.

@article{liang2024diffusion4d,
  title={Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models},
  author={Liang, Hanwen and Yin, Yuyang and Xu, Dejia and Liang, Hanxue and Wang, Zhangyang and Plataniotis, Konstantinos N and Zhao, Yao and Wei, Yunchao},
  journal={arXiv preprint arXiv:2405.16645},
  year={2024}
}