diffusers-internal-dev/chronoedit-modular

1---2library_name: diffusers3tags:4- modular_diffusers5---6 7# Modular ChronoEdit8 9Modular implementation of [`nvidia/ChronoEdit-14B-Diffusers`](https://hf.co/nvidia/ChronoEdit-14B-Diffusers).10 11## Code12 13<details>14<summary>Unfold</summary>15 16```py17"""18Mimicked from https://huggingface.co/spaces/nvidia/ChronoEdit/blob/main/app.py19"""20 21from diffusers.modular_pipelines import WanModularPipeline, ModularPipelineBlocks22from diffusers.utils import load_image23from diffusers import UniPCMultistepScheduler24import torch25from PIL import Image26 27repo_id = "diffusers-internal-dev/chronoedit-modular"28blocks = ModularPipelineBlocks.from_pretrained(repo_id, trust_remote_code=True)29pipe = WanModularPipeline(blocks, repo_id)30pipe.load_components(31    trust_remote_code=True,32    device_map="cuda",33    torch_dtype={"default": torch.bfloat16, "image_encoder": torch.float32},34)35pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=2.0)36pipe.load_lora_weights("nvidia/ChronoEdit-14B-Diffusers", weight_name="lora/chronoedit_distill_lora.safetensors")37pipe.fuse_lora(lora_scale=1.0)38 39image = load_image("https://huggingface.co/spaces/nvidia/ChronoEdit/resolve/main/examples/3.png")40prompt = "Transform the image so that inside the floral teacup of steaming tea, a small, cute mouse is sitting and taking a bath; the mouse should look relaxed and cheerful, with a tiny white bath towel draped over its head as if enjoying a spa moment, while the steam rises gently around it, blending seamlessly with the warm and cozy atmosphere."41 42# image is resized within the pipeline unlike https://huggingface.co/spaces/nvidia/ChronoEdit/blob/main/app.py#L15143# refer to `ChronoEditImageInputStep`.44out = pipe(45    image=image,46    prompt=prompt,  # todo: enhance prompt47    num_inference_steps=8,  # todo: implement temporal reasoning48    num_frames=5,  # https://huggingface.co/spaces/nvidia/ChronoEdit/blob/main/app.py#L15249    output_type="np",50    generator=torch.manual_seed(0),51)52frames = out.values["videos"][0]53Image.fromarray((frames[-1] * 255).clip(0, 255).astype("uint8")).save("demo.png")54```55 56</details>57 58You can find it [here](./example.py) too.59 60> [!TIP]61> Make sure `diffusers` is installed from source: `pip install git+https://github.com/huggingface/diffusers`.62 63## Results64 65<table>66  <tr>67    <td><img src="https://huggingface.co/spaces/nvidia/ChronoEdit/resolve/main/examples/3.png" alt="First Image"></td>68    <td><img src="./demo.png" alt="Edited Image"></td>69  </tr>70  <caption><i>Transform the image so that inside the floral teacup of steaming tea, a small, cute mouse is sitting and taking a bath; the mouse should look relaxed and cheerful, with a tiny white bath towel draped over its head as if enjoying a spa moment, while the steam rises gently around it, blending seamlessly with the warm and cozy atmosphere</i>.</caption>71</table>72 73## Notes74 751. This implementation doesn't have temporal reasoning.762. This doesn't use a separate prompt enhancer model.