codingrobot/simpletuner-lora

1---2license: other3base_model: "black-forest-labs/FLUX.1-dev"4tags:5  - flux6  - flux-diffusers7  - text-to-image8  - image-to-image9  - diffusers10  - simpletuner11  - not-for-all-audiences12  - lora13 14  - template:sd-lora15  - standard16pipeline_tag: text-to-image17inference: true18widget:19- text: 'unconditional (blank prompt)'20  parameters:21    negative_prompt: 'blurry, cropped, ugly'22  output:23    url: ./assets/image_0_0.png24- text: 'a breathtaking anime-style portrait of nikolai, capturing his essence with vibrant colors and expressive features'25  parameters:26    negative_prompt: 'blurry, cropped, ugly'27  output:28    url: ./assets/image_1_0.png29- text: 'a high-quality, detailed photograph of nikolai as a sous-chef, immersed in the art of culinary creation'30  parameters:31    negative_prompt: 'blurry, cropped, ugly'32  output:33    url: ./assets/image_2_0.png34- text: 'a lifelike and intimate portrait of nikolai, showcasing his unique personality and charm'35  parameters:36    negative_prompt: 'blurry, cropped, ugly'37  output:38    url: ./assets/image_3_0.png39- text: 'a cinematic, visually stunning photo of nikolai, emphasizing his dramatic and captivating presence'40  parameters:41    negative_prompt: 'blurry, cropped, ugly'42  output:43    url: ./assets/image_4_0.png44- text: 'an elegant and timeless portrait of nikolai, exuding grace and sophistication'45  parameters:46    negative_prompt: 'blurry, cropped, ugly'47  output:48    url: ./assets/image_5_0.png49- text: 'a dynamic and adventurous photo of nikolai, captured in an exciting, action-filled moment'50  parameters:51    negative_prompt: 'blurry, cropped, ugly'52  output:53    url: ./assets/image_6_0.png54- text: 'a mysterious and enigmatic portrait of nikolai, shrouded in shadows and intrigue'55  parameters:56    negative_prompt: 'blurry, cropped, ugly'57  output:58    url: ./assets/image_7_0.png59- text: 'a vintage-style portrait of nikolai, evoking the charm and nostalgia of a bygone era'60  parameters:61    negative_prompt: 'blurry, cropped, ugly'62  output:63    url: ./assets/image_8_0.png64- text: 'an artistic and abstract representation of nikolai, blending creativity with visual storytelling'65  parameters:66    negative_prompt: 'blurry, cropped, ugly'67  output:68    url: ./assets/image_9_0.png69- text: 'a futuristic and cutting-edge portrayal of nikolai, set against a backdrop of advanced technology'70  parameters:71    negative_prompt: 'blurry, cropped, ugly'72  output:73    url: ./assets/image_10_0.png74- text: 'A picture of nikolai'75  parameters:76    negative_prompt: 'blurry, cropped, ugly'77  output:78    url: ./assets/image_11_0.png79---80 81# simpletuner-lora82 83This is a PEFT LoRA derived from [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev).84 85The main validation prompt used during training was:86```87A picture of nikolai88```89 90 91## Validation settings92- CFG: `3.0`93- CFG Rescale: `0.0`94- Steps: `20`95- Sampler: `FlowMatchEulerDiscreteScheduler`96- Seed: `42`97- Resolution: `1024x1024`98- Skip-layer guidance: 99 100Note: The validation settings are not necessarily the same as the [training settings](#training-settings).101 102You can find some example images in the following gallery:103 104 105<Gallery />106 107The text encoder **was not** trained.108You may reuse the base model text encoder for inference.109 110 111## Training settings112 113- Training epochs: 29114- Training steps: 500115- Learning rate: 0.0001116  - Learning rate schedule: polynomial117  - Warmup steps: 100118- Max grad value: 1.0119- Effective batch size: 1120  - Micro-batch size: 1121  - Gradient accumulation steps: 1122  - Number of GPUs: 1123- Gradient checkpointing: True124- Prediction type: flow_matching (extra parameters=['shift=3', 'flux_guidance_mode=constant', 'flux_guidance_value=1.0', 'flux_lora_target=mmdit'])125- Optimizer: adamw_bf16126- Trainable parameter precision: Pure BF16127- Base model precision: `fp8-torchao`128- Caption dropout probability: 0.1%129 130 131- LoRA Rank: 16132- LoRA Alpha: None133- LoRA Dropout: 0.1134- LoRA initialisation style: default135- LoRA mode: Standard136    137 138## Datasets139 140### dreambooth-subject141- Repeats: 0142- Total number of images: 17143- Total number of aspect buckets: 1144- Resolution: 1.048576 megapixels145- Cropped: False146- Crop style: None147- Crop aspect: None148- Used for regularisation data: No149 150 151## Inference152 153 154```python155import torch156from diffusers import DiffusionPipeline157 158model_id = 'black-forest-labs/FLUX.1-dev'159adapter_id = 'codingrobot/simpletuner-lora'160pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16161pipeline.load_lora_weights(adapter_id)162 163prompt = "A picture of nikolai"164 165 166## Optional: quantise the model to save on vram.167## Note: The model was quantised during training, and so it is recommended to do the same during inference time.168from optimum.quanto import quantize, freeze, qint8169quantize(pipeline.transformer, weights=qint8)170freeze(pipeline.transformer)171    172pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level173model_output = pipeline(174    prompt=prompt,175    num_inference_steps=20,176    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),177    width=1024,178    height=1024,179    guidance_scale=3.0,180).images[0]181 182model_output.save("output.png", format="PNG")183 184```185 186 187 188