SDXL LoRA DreamBooth - AdamLucek/sdxl-base-1.0-greenchair-dreambooth-lora

Model description

These are LoRA DreamBooth weights for stabilityai/stable-diffusion-xl-base-1.0.

The weights were trained using DreamBooth on the AdamLucek/green-chair Dataset.

LoRA for the text encoder was enabled: True.

Special VAE used for training: madebyollin/sdxl-vae-fp16-fix.

Trigger words

You should use a photo of sks chair to trigger the image generation.

Example Generations

Reference Image Generated Images on a street in new york, in a desert, in a jungle, in the color blue

Download model

Weights for this model are available in Safetensors format.

Intended uses & limitations

COLAB Notebook Here

How to use

# Load Stable Diffusion XL Base 1.0
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", 
    torch_dtype=torch.float16,
    variant="fp16", 
    use_safetensors=True
).to("cuda")

# Optional, enable cpu offloading
pipe.enable_model_cpu_offload()

# Load LoRA Weights
pipe.load_lora_weights("AdamLucek/sdxl-base-1.0-greenchair-dreambooth-lora")

# Generate an Image
image = pipe(
    prompt = "a photo of sks chair",
    num_inference_steps=50,
    height=1024,
    width=1024,
).images[0]

# Save the Image
image.save("green_chair.png")

Limitations and bias

Note: Limited tuning of hyperparameters

Note: See original Stable Diffusion XL Base 1.0 page for additional limitations and biases

Training details

Video Overview

Trained using Dreambooth via Diffusers on a single A100

Training Script:

accelerate launch train_dreambooth_lora_sdxl.py \
  --pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0"  \
  --dataset_name="AdamLucek/green-chair" \
  --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
  --output_dir="lora-trained-xl" \
  --train_text_encoder \
  --instance_prompt="a photo of sks chair" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks chair in an apartment" \
  --validation_epochs=25 \
  --seed="0" \
  --hub_model_id="sdxl-base-1.0-greenchair-dreambooth-lora" \
  --push_to_hub