LoRA text2image fine-tuning - ra100/sdxl-lora-lower-decks-aesthetic

These are LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0. The weights were fine-tuned on the None dataset.

Tokens to use:

• stld aesthetic - Star Trek Lower Decks style
• blue and gray stldunf uniform - to see uniform
• nacelles, deflector, saucer - to have more Federation style ships

Images below generated with prompt:

a female in a blue and gray stldunf uniform standing in woods next to a lake, stld aesthetic

You can find some example images in the following.

"a female in a blue and gray stldunf uniform standing in woods next to a lake, stld aesthetic"

"man in yellow and gray stldunf uniform sitting, on spaceship, stld aesthetic"

"a cat, stld aesthetic"

"red planet with green moon, space scenery, stars in background, nebulae, stld aesthetic"

"Luna Lovegood in a red and gray stldunf uniform standing in scifi sickbay, stld aesthetic"

"space ship with nacelles, saucer, deflector, floating in space above planet, stars, stld aestehtic"

LoRA for the text encoder was enabled: True.

Special VAE used for training: madebyollin/sdxl-vae-fp16-fix.

Training command:

export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"
export OUTPUT_DIR="stld-aesthetic/model47-grd16-1e4-fp16"
export MODEL_VAE="madebyollin/sdxl-vae-fp16-fix"
export DATASET_DIR="stld-aesthetic/img47"

export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512

accelerate launch diffusers/examples/text_to_image/train_text_to_image_lora_sdxl.py \
  --output_dir=$OUTPUT_DIR \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --pretrained_vae_model_name_or_path=$MODEL_VAE \
  --train_data_dir=$DATASET_DIR \
  --hub_model_id="ra100/sdxl-lora-lower-decks-aesthetic" \
  --checkpointing_steps=1000 \
  --checkpoints_total_limit=10 \
  --learning_rate=1e-4 \
  --lr_scheduler="constant" \
  --mixed_precision="fp16" \
  --lr_warmup_steps=0 \
  --max_train_steps=4000 \
  --num_validation_images=4 \
  --resolution=1024 \
  --seed="46158136" \
  --train_batch_size=1 \
  --train_text_encoder \
  --report_to="wandb" \
  --validation_epochs=30 \
  --gradient_accumulation_steps=16 \
  --gradient_checkpointing \
  --allow_tf32 \
  --validation_prompt="a female in a blue and gray stldunf uniform standing in woods next to a lake, stld aesthetic"