Modelpublic

jasperai/Flux.1-dev-Controlnet-Surface-Normals

sourceHugging Faceotherupdated 2y agoView on Hugging Face

101likes557downloads

Model Card

⚡ Flux.1-dev: Surface Normals ControlNet ⚡

This is Flux.1-dev ControlNet for Surface Normals map developed by Jasper research team.

How to use

This model can be used directly with the diffusers library

python

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetModel
from diffusers.pipelines import FluxControlNetPipeline

# Load pipeline
controlnet = FluxControlNetModel.from_pretrained(
  "jasperai/Flux.1-dev-Controlnet-Surface-Normals",
  torch_dtype=torch.bfloat16
)
pipe = FluxControlNetPipeline.from_pretrained(
  "black-forest-labs/FLUX.1-dev",
  controlnet=controlnet,
  torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Load a control image
control_image = load_image(
  "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/surface.jpg"
)

prompt = "a man showing stop sign in front of window"

image = pipe(
    prompt, 
    control_image=control_image,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=28, 
    guidance_scale=3.5,
    height=control_image.size[1],
    width=control_image.size[0]
).images[0]
image

💡 Note: You can compute the conditioning map using the NormalBaeDetector from the controlnet_aux library

python

from controlnet_aux import NormalBaeDetector
from diffusers.utils import load_image 

normal_bae = NormalBaeDetector.from_pretrained("lllyasviel/Annotators")

normal_bae.to("cuda")

# Load an image
im = load_image(
  "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/output.jpg"
)

surface = normal_bae(im)

Training

This model was trained with surface normals maps computed with Clipdrop's surface normals estimator model as well as an open-souce surface normals estimation model such as Boundary Aware Encoder (BAE).

Licence

This model falls under the Flux.1-dev model licence.