DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth
0
1---2license: apache-2.03---4# Qwen-Image Image Structure Control Model - Depth ControlNet5 67 8## Model Introduction9 10This model is an image structure control model based on [Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image), with a ControlNet architecture that enables structural control of generated images using depth maps. The training framework is built upon [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), and the dataset used for training is [BLIP3o](https://modelscope.cn/datasets/BLIP3o/BLIP3o-60k).11 12## Result Demonstration13 14|Depth Map|Generated Image 1|Generated Image 2|15|-|-|-|16||||17||||18||||19 20## Inference Code21```22git clone https://github.com/modelscope/DiffSynth-Studio.git 23cd DiffSynth-Studio24pip install -e .25```26 27```python28from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig, ControlNetInput29from PIL import Image30import torch31from modelscope import dataset_snapshot_download32 33 34pipe = QwenImagePipeline.from_pretrained(35 torch_dtype=torch.bfloat16,36 device="cuda",37 model_configs=[38 ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),39 ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),40 ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),41 ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth", origin_file_pattern="model.safetensors"),42 ],43 tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"),44)45 46dataset_snapshot_download(47 dataset_id="DiffSynth-Studio/example_image_dataset",48 local_dir="./data/example_image_dataset",49 allow_file_pattern="depth/image_1.jpg"50)51 52controlnet_image = Image.open("data/example_image_dataset/depth/image_1.jpg").resize((1328, 1328))53```54 55prompt = "Exquisite portrait, underwater girl, flowing blue dress, gently floating hair, translucent lighting, surrounded by bubbles, serene expression, intricate details, dreamy and ethereal."56image = pipe(57 prompt, seed=0,58 blockwise_controlnet_inputs=[ControlNetInput(image=controlnet_image)]59)60image.save("image.jpg")