|
--- |
|
license: creativeml-openrail-m |
|
base_model: runwayml/stable-diffusion-v1-5 |
|
tags: |
|
- stable-diffusion |
|
- stable-diffusion-diffusers |
|
- text-to-image |
|
- diffusers |
|
- controlnet |
|
- jax-diffusers-event |
|
inference: true |
|
datasets: |
|
- mfidabel/sam-coyo-2k |
|
- mfidabel/sam-coyo-2.5k |
|
- mfidabel/sam-coyo-3k |
|
language: |
|
- en |
|
library_name: diffusers |
|
--- |
|
|
|
# ControlNet - mfidabel/controlnet-segment-anything |
|
|
|
These are controlnet weights trained on runwayml/stable-diffusion-v1-5 with a new type of conditioning. You can find some example images in the following. |
|
|
|
**prompt**: contemporary living room of a house |
|
|
|
**negative prompt**: low quality |
|
![images_0)](./images_0.png) |
|
|
|
**prompt**: new york buildings, Vincent Van Gogh starry night |
|
|
|
**negative prompt**: low quality, monochrome |
|
![images_1)](./images_1.png) |
|
|
|
**prompt**: contemporary living room, high quality, 4k, realistic |
|
|
|
**negative prompt**: low quality, monochrome, low res |
|
![images_2)](./images_2.png) |
|
|
|
## Training |
|
|
|
**Training Data** This model was trained using a Segmented dataset based on the [COYO-700M Dataset](https://huggingface.co/datasets/kakaobrain/coyo-700m). |
|
[Stable Diffusion v1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5) checkpoint was used as the base model for the controlnet. |
|
|
|
|
|
The model was trained as follows: |
|
|
|
- 25k steps with the [SAM-COYO-2k](https://huggingface.co/datasets/mfidabel/sam-coyo-2k) dataset |
|
- 28k steps with the [SAM-COYO-2.5k](https://huggingface.co/datasets/mfidabel/sam-coyo-2.5k) dataset |
|
- 38k steps with the [SAM-COYO-3k](https://huggingface.co/datasets/mfidabel/sam-coyo-3k) dataset |
|
|
|
In that particular order. |
|
|
|
- **Hardware**: Google Cloud TPUv4-8 VM |
|
|
|
- **Optimizer**: AdamW |
|
|
|
- **Train Batch Size**: 2 x 4 = 8 |
|
|
|
- **Learning rate**: 0.00001 constant |
|
|
|
- **Gradient Accumulation Steps**: 1 |
|
|
|
- **Resolution**: 512 |
|
|
|
|