manycore-research/FLUX.1-Layout-ControlNet

FLUX.1-Layout-ControlNet for SpatialGen

FLUX.1-Layout-ControlNet is a key component of the SpatialGen. FLUX.1 Layout Controlnet [dev] is a ControlNet conditioned by a semantic image. It is capable of generating a 2D image based on a text description while following the semantic image layout. This ControlNet is applicable to [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev) and is leveraged within the SpatialGen framework for 3D scene synthesis.

• —📖 Paper: SPATIALGEN: Layout-guided 3D Indoor Scene Generation
• —🌐 Project Page
• —💻 Code Repository

Showcase

<div style="display: grid; grid-template-columns: repeat(2, 1fr); gap: 20px; justify-items: center;">

<div> <img src="https://cdn-uploads.huggingface.co/production/uploads/6437c0ead38ce48bdd4b0067/1pQtOX6Dwzg3dIlCg2mag.png" width="100%" style="display: inline-block;" /> <br> <p><i>This serene Chinese-inspired bedroom with a traditional bed frame featuring intricate carvings. Use soft, muted colors for the bedding and walls. Hang delicate, calligraphic artworks above the bed. Place a refined wooden nightstand with a circular mirror beside the bed and a built-in wardrobe with sliding doors. Large windows should be adorned with lightweight, embroidered curtains that allow natural light to gently enter the room on the wooden board.</i></p> </div>

<div> <img src="https://cdn-uploads.huggingface.co/production/uploads/6437c0ead38ce48bdd4b0067/MzcDCHKCKlBwGZeg-dZHl.png" width="100%" style="display: inline-block;" /> <br> <p><i>This elegant European-style bedroom features a luxurious bed with a tufted headboard, adorned with fine linens. Incorporate classic abstract paintings above the bed and a sophisticated floating nightstand with a decorative mirror. Include a built-in wardrobe with ornate details and large windows draped with sheer, flowing curtains that let in soft, diffused light.</i></p> </div>

<div> <img src="https://cdn-uploads.huggingface.co/production/uploads/6437c0ead38ce48bdd4b0067/wrxvo9cNWq1otXixeK6gf.png" width="100%" style="display: inline-block;" /> <br> <p><i>This whimsical cartoon bedroom with a playful bed that has a fun, tufted headboard. The bedding should feature bright, cheerful patterns. Hang colorful, abstract cartoons on the walls above the bed. Include a quirky, floating nightstand with a round, animated mirror and a built-in wardrobe with open shelves filled with toys. Large windows with sheer, patterned curtains should let in plenty of light, creating a lively atmosphere.</i></p> </div>

<div> <img src="https://cdn-uploads.huggingface.co/production/uploads/6437c0ead38ce48bdd4b0067/CHtrW7PRM0_wHDFmnnnYa.png" width="100%" style="display: inline-block;" /> <br> <p><i>This futuristic cyberpunk bedroom with a sleek, high-tech bed featuring a digital display headboard. The bedding should have a metallic sheen with neon accents. Install holographic abstract art above the bed and a floating nightstand with a reflective, circular mirror. Add a built-in wardrobe with illuminated shelves and large windows with smart, translucent curtains that filter the city lights. The room should exude a cool, tech-savvy vibe.</i></p> </div>

</div>

Diffusers

To use FLUX.1-Layout-Controlnet with the 🧨 diffusers python library, first install or upgrade diffusers, peft.

pip install -U git+https://github.com/huggingface/diffusers
pip install -U peft

Then you can use the FluxControlNetPipeline to run it:

import torch
from diffusers import FluxControlNetModel
from diffusers.pipelines import FluxControlNetPipeline
from diffusers.utils import load_image

generator = torch.Generator(device="cuda").manual_seed(87544357)

controlnet = FluxControlNetModel.from_pretrained(
  "manycore-research/FLUX.1-Layout-ControlNet",
  torch_dtype=torch.bfloat16,
)
pipe = FluxControlNetPipeline.from_pretrained(
  "black-forest-labs/FLUX.1-dev/",
  controlnet=controlnet,
  torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

control_image = load_image("https://huggingface.co/manycore-research/FLUX.1-Layout-ControlNet/blob/main/assets/input.png")
prompt = "This elegant European-style bedroom features a luxurious bed with a tufted headboard, adorned with fine linens. Incorporate classic abstract paintings above the bed and a sophisticated floating nightstand with a decorative mirror. Include a built-in wardrobe with ornate details and large windows draped with sheer, flowing curtains that let in soft, diffused light."

image = pipe(
    prompt,
    control_image=control_image,
    controlnet_conditioning_scale=0.7,
    num_inference_steps=25,
    guidance_scale=7,
    height=1024,
    width=1024,
    generator=generator,
    num_images_per_prompt=1,
).images[0]
image.save("output.png")

LICENSE

FLUX.1-Layout-ControlNet is licensed under the FLUX.1-dev Non-Commercial License.