diffusers-internal-dev/sd35-controlnet-depth-8b
150
1---2license: other3license_name: stabilityai-ai-community4license_link: LICENSE.md5tags:6- stable-diffusion7- controlnet8inference: true9extra_gated_prompt: >-10 By clicking "Agree", you agree to the [License11 Agreement](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)12 and acknowledge Stability AI's [Privacy13 Policy](https://stability.ai/privacy-policy).14extra_gated_fields:15 Name: text16 Email: text17 Country: country18 Organization or Affiliation: text19 Receive email updates and promotions on Stability AI products, services, and research?:20 type: select21 options:22 - 'Yes'23 - 'No'24 What do you intend to use the model for?: 25 type: select26 options:27 - Research28 - Personal use29 - Creative Professional30 - Startup31 - Enterprise32 I agree to the License Agreement and acknowledge Stability AI's Privacy Policy: checkbox33 34language:35- en36pipeline_tag: text-to-image37---38 39# Stable Diffusion 3.5 ControlNets4041 42## Model43 44This repository provides a number of ControlNet models trained for use with [Stable Diffusion 3.5 Large](https://stability.ai/news/introducing-stable-diffusion-3-5).45 46The following control types are available:47- Canny - Use a Canny edge map to guide the structure of the generated image. This is especially useful for illustrations, but works with all styles.48- Depth - use a depth map, generated by DepthFM, to guide generation. Some example use cases include generating architectural renderings, or texturing 3D assets.49- Blur - can be used to perform extremely high fidelity upscaling. A common use case is to tile an input image, apply the ControlNet to each tile, and merge the tiles to produce a higher resolution image. A more in-depth description of this use case is [here](https://github.com/lllyasviel/ControlNet-v1-1-nightly/issues/125).50 - For Comfy users, [this extension](https://github.com/shiimizu/ComfyUI-TiledDiffusion) provides support.51 - We recommend tiling the image at a tile size between 128 and 512.52 53All currently released ControlNets are compatible only with Stable Diffusion 3.5 Large (8b).54 55Additional ControlNet models, including 2B versions of the variants above, and multiple other control types, will be added to this repository in the future.56 57Please note: This model is released under the [Stability Community License](https://stability.ai/community-license-agreement). Visit [Stability AI](https://stability.ai/license) to learn or [contact us](https://stability.ai/enterprise) for commercial licensing details.58 59 60### License61 62Here are the key components of the license:63* Free for non-commercial use: Individuals and organizations can use the model free of charge for non-commercial use, including scientific research. 64* Free for commercial use (up to $1M in annual revenue): Startups, small to medium-sized businesses, and creators can use the model for commercial purposes at no cost, as long as their total annual revenue is less than $1M.65* Ownership of outputs: Retain ownership of the media generated without restrictive licensing implications.66 67For organizations with annual revenue more than $1M, please contact us [here](https://stability.ai/enterprise) to inquire about an Enterprise License.68 69## Usage70 71For local or self-hosted use, we recommend [ComfyUI](https://github.com/comfyanonymous/ComfyUI) for node-based UI inference, or the [standalone SD3.5 repo](https://github.com/Stability-AI/sd3.5) for programmatic use.72 73You can also use [🧨 Diffusers](https://github.com/huggingface/diffusers).74 75### Usage in ComfyUI76 77Please see the [ComfyUI announcement blog post](http://blog.comfy.org/sd3-5-large-controlnet/) for details on usage within Comfy, including example workflows.78 79### Usage in SD3.5 Standalone Repo80Install the repo:81```82git clone git@github.com:Stability-AI/sd3.5.git83pip install -r requirements.txt84```85 86Then, download the models and sample images like so:87 88```89input/canny.png90models/clip_g.safetensors91models/clip_l.safetensors92models/t5xxl.safetensors93models/sd3.5_large.safetensors94models/sd3.5_large_controlnet_canny.safetensors95```96 97and then you can run98```99python sd3_infer.py --controlnet_ckpt models/sd3.5_large_controlnet_canny.safetensors --controlnet_cond_image input/canny.png --prompt "An adorable fluffy pastel creature"100```101 102Which should give you an image like below:103 104105 106The conditioning image should already be preprocessed before being used as input to the standalone repo; sd3.5 does not implement the preprocessing code below.107 108### Usage in Diffusers 🧨109 110Make sure you upgrade to the latest version of diffusers: `pip install -U diffusers`. And then you can run:111 112```python113import torch114from diffusers import StableDiffusion3ControlNetPipeline, SD3ControlNetModel115from diffusers.utils import load_image116 117controlnet = SD3ControlNetModel.from_pretrained("diffusers-internal-dev/sd35-controlnet-depth-8b", torch_dtype=torch.float16)118pipe = StableDiffusion3ControlNetPipeline.from_pretrained(119 "stabilityai/stable-diffusion-3.5-large",120 controlnet=controlnet,121 torch_dtype=torch.float16,122).to("cuda")123 124control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_depth.png")125generator = torch.Generator(device="cpu").manual_seed(0)126image = pipe(127 prompt = "a photo of a man", 128 control_image=control_image, 129 guidance_scale=4.5,130 num_inference_steps=40,131 generator=generator,132 max_sequence_length=77,133).images[0]134image.save('depth-8b.jpg')135```136 137### Preprocessing138 139Below are code snippets for preprocessing the various control image types.140 141#### Canny142 143```python144import torchvision.transforms.functional as F145# assuming img is a PIL image146img = F.to_tensor(img)147img = cv2.cvtColor(img.transpose(1, 2, 0), cv2.COLOR_RGB2GRAY)148img = cv2.Canny(img, 100, 200)149```150 151#### Blur152```python153import torchvision.transforms as transforms154# assuming img is a PIL image155gaussian_blur = transforms.GaussianBlur(kernel_size=50)156blurred_image = gaussian_blur(img)157```158 159#### Depth160 161```python162# install image_gen_aux with: pip install git+https://github.com/asomoza/image_gen_aux.git163from image_gen_aux import DepthPreprocessor164 165image = load_image("path to image")166 167depth_preprocessor = DepthPreprocessor.from_pretrained("depth-anything/Depth-Anything-V2-Large-hf").to("cuda")168depth_image = depth_preprocessor(image, invert=True)[0].convert("RGB")169```170 171### Tips172- We recommend starting with a ControlNet strength of 0.7-0.8, and adjusting as needed.173- Euler sampler and a slightly higher step count (50-60) gives best results, especially with Canny.174- Pass `--text_encoder_device <device_name>` to load the text encoders directly to VRAM, which can speed up the full inference loop at the cost of extra VRAM usage.175 176## Uses177 178All uses of the model must be in accordance with our [Acceptable Use Policy](https://stability.ai/use-policy).179 180### Out-of-Scope Uses181 182The model was not trained to be factual or true representations of people or events. As such, using the model to generate such content is out-of-scope of the abilities of this model.183 184### Training Data and Strategy185 186These models were trained on a wide variety of data, including synthetic data and filtered publicly available data.187 188## Safety189 190We believe in safe, responsible AI practices and take deliberate measures to ensure Integrity starts at the early stages of development. This means we have taken and continue to take reasonable steps to prevent the misuse of Stable Diffusion 3.5 by bad actors. For more information about our approach to Safety please visit our [Safety page](https://stability.ai/safety).191 192### Integrity Evaluation193 194Our integrity evaluation methods include structured evaluations and red-teaming testing for certain harms. Testing was conducted primarily in English and may not cover all possible harms. 195 196### Risks identified and mitigations:197 198* Harmful content: We have implemented safeguards that attempt to strike the right balance between usefulness and preventing harm. However, this does not guarantee that all possible harmful content has been removed. All developers and deployers should exercise caution and implement content safety guardrails based on their specific product policies and application use cases.199* Misuse: Technical limitations and developer and end-user education can help mitigate against malicious applications of models. All users are required to adhere to our Acceptable Use Policy, including when applying fine-tuning and prompt engineering mechanisms. Please reference the Stability AI Acceptable Use Policy for information on violative uses of our products.200* Privacy violations: Developers and deployers are encouraged to adhere to privacy regulations with techniques that respect data privacy.201 202### Acknowledgements203 204 205- Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, authors of the original [ControlNet paper](https://arxiv.org/abs/2302.05543).206- Lvmin Zhang, who also developed the [Tile ControlNet](https://huggingface.co/lllyasviel/control_v11f1e_sd15_tile), which inspired the Blur ControlNet.207- [Diffusers](https://github.com/huggingface/diffusers) library authors, whose code was referenced during development.208- [InstantX](https://github.com/instantX-research) team, whose Flux and SD3 ControlNets were also referenced during training.209- All early testers and raters of the models, and the Stability AI team.210 211### Contact212 213Please report any issues with the model or contact us:214 215* Safety issues: safety@stability.ai216* Security issues: security@stability.ai217* Privacy issues: privacy@stability.ai218* License and general: https://stability.ai/license219* Enterprise license: https://stability.ai/enterprise