Duplicate from TencentARC/t2i-adapter-sketch-sdxl-1.0
Browse filesCo-authored-by: Suraj Patil <valhalla@users.noreply.huggingface.co>
- .gitattributes +35 -0
- README.md +133 -0
- config.json +15 -0
- diffusion_pytorch_model.fp16.safetensors +3 -0
- diffusion_pytorch_model.safetensors +3 -0
.gitattributes
ADDED
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
base_model: stabilityai/stable-diffusion-xl-base-1.0
|
4 |
+
tags:
|
5 |
+
- art
|
6 |
+
- t2i-adapter
|
7 |
+
- image-to-image
|
8 |
+
- stable-diffusion-xl-diffusers
|
9 |
+
- stable-diffusion-xl
|
10 |
+
---
|
11 |
+
|
12 |
+
# T2I-Adapter-SDXL - Sketch
|
13 |
+
|
14 |
+
T2I Adapter is a network providing additional conditioning to stable diffusion. Each t2i checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint.
|
15 |
+
|
16 |
+
This checkpoint provides conditioning on sketch for the StableDiffusionXL checkpoint. This was a collaboration between **Tencent ARC** and [**Hugging Face**](https://huggingface.co/).
|
17 |
+
|
18 |
+
## Model Details
|
19 |
+
- **Developed by:** T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
|
20 |
+
- **Model type:** Diffusion-based text-to-image generation model
|
21 |
+
- **Language(s):** English
|
22 |
+
- **License:** Apache 2.0
|
23 |
+
- **Resources for more information:** [GitHub Repository](https://github.com/TencentARC/T2I-Adapter), [Paper](https://arxiv.org/abs/2302.08453).
|
24 |
+
- **Model complexity:**
|
25 |
+
| | SD-V1.4/1.5 | SD-XL | T2I-Adapter | T2I-Adapter-SDXL |
|
26 |
+
| --- | --- |--- |--- |--- |
|
27 |
+
| Parameters | 860M | 2.6B |77 M | 77/79 M | |
|
28 |
+
- **Cite as:**
|
29 |
+
|
30 |
+
@misc{
|
31 |
+
title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models},
|
32 |
+
author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie},
|
33 |
+
year={2023},
|
34 |
+
eprint={2302.08453},
|
35 |
+
archivePrefix={arXiv},
|
36 |
+
primaryClass={cs.CV}
|
37 |
+
}
|
38 |
+
|
39 |
+
### Checkpoints
|
40 |
+
|
41 |
+
| Model Name | Control Image Overview| Control Image Example | Generated Image Example |
|
42 |
+
|---|---|---|---|
|
43 |
+
|[TencentARC/t2i-adapter-canny-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-canny-sdxl-1.0)<br/> *Trained with canny edge detection* | A monochrome image with white edges on a black background.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_canny.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_canny.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_canny.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_canny.png"/></a>|
|
44 |
+
|[TencentARC/t2i-adapter-sketch-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-sketch-sdxl-1.0)<br/> *Trained with [PidiNet](https://github.com/zhuoinoulu/pidinet) edge detection* | A hand-drawn monochrome image with white outlines on a black background.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_sketch.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_sketch.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_sketch.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_sketch.png"/></a>|
|
45 |
+
|[TencentARC/t2i-adapter-lineart-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0)<br/> *Trained with lineart edge detection* | A hand-drawn monochrome image with white outlines on a black background.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_lin.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_lin.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_lin.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_lin.png"/></a>|
|
46 |
+
|[TencentARC/t2i-adapter-depth-midas-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0)<br/> *Trained with Midas depth estimation* | A grayscale image with black representing deep areas and white representing shallow areas.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_depth_mid.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_depth_mid.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_depth_mid.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_depth_mid.png"/></a>|
|
47 |
+
|[TencentARC/t2i-adapter-depth-zoe-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-depth-zoe-sdxl-1.0)<br/> *Trained with Zoe depth estimation* | A grayscale image with black representing deep areas and white representing shallow areas.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_depth_zeo.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_depth_zeo.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_depth_zeo.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_depth_zeo.png"/></a>|
|
48 |
+
|[TencentARC/t2i-adapter-openpose-sdxl-1.0](https://huggingface.co/TencentARC/t2i-adapter-openpose-sdxl-1.0)<br/> *Trained with OpenPose bone image* | A [OpenPose bone](https://github.com/CMU-Perceptual-Computing-Lab/openpose) image.|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/openpose.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/openpose.png"/></a>|<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/res_pose.png"><img width="64" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/res_pose.png"/></a>|
|
49 |
+
|
50 |
+
|
51 |
+
## Demo:
|
52 |
+
|
53 |
+
Try out the model with your own hand-drawn sketches/doodles in the [Doodly Space](https://huggingface.co/spaces/TencentARC/T2I-Adapter-SDXL-Sketch)!
|
54 |
+
|
55 |
+
![app image](https://huggingface.co/datasets/valhalla/images/resolve/main/Screenshot%202023-09-08%20at%2011.03.36%20AM.png)
|
56 |
+
|
57 |
+
|
58 |
+
## Example
|
59 |
+
|
60 |
+
To get started, first install the required dependencies:
|
61 |
+
|
62 |
+
```bash
|
63 |
+
pip install -U git+https://github.com/huggingface/diffusers.git
|
64 |
+
pip install -U controlnet_aux==0.0.7 # for conditioning models and detectors
|
65 |
+
pip install transformers accelerate safetensors
|
66 |
+
```
|
67 |
+
|
68 |
+
1. Images are first downloaded into the appropriate *control image* format.
|
69 |
+
2. The *control image* and *prompt* are passed to the [`StableDiffusionXLAdapterPipeline`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/t2i_adapter/pipeline_stable_diffusion_xl_adapter.py#L125).
|
70 |
+
|
71 |
+
Let's have a look at a simple example using the [Canny Adapter](https://huggingface.co/TencentARC/t2i-adapter-lineart-sdxl-1.0).
|
72 |
+
|
73 |
+
- Dependency
|
74 |
+
```py
|
75 |
+
from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler, AutoencoderKL
|
76 |
+
from diffusers.utils import load_image, make_image_grid
|
77 |
+
from controlnet_aux.pidi import PidiNetDetector
|
78 |
+
import torch
|
79 |
+
|
80 |
+
# load adapter
|
81 |
+
adapter = T2IAdapter.from_pretrained(
|
82 |
+
"TencentARC/t2i-adapter-sketch-sdxl-1.0", torch_dtype=torch.float16, varient="fp16"
|
83 |
+
).to("cuda")
|
84 |
+
|
85 |
+
# load euler_a scheduler
|
86 |
+
model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
|
87 |
+
euler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
|
88 |
+
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
|
89 |
+
pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
|
90 |
+
model_id, vae=vae, adapter=adapter, scheduler=euler_a, torch_dtype=torch.float16, variant="fp16",
|
91 |
+
).to("cuda")
|
92 |
+
pipe.enable_xformers_memory_efficient_attention()
|
93 |
+
|
94 |
+
pidinet = PidiNetDetector.from_pretrained("lllyasviel/Annotators").to("cuda")
|
95 |
+
```
|
96 |
+
|
97 |
+
- Condition Image
|
98 |
+
```py
|
99 |
+
url = "https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_sketch.png"
|
100 |
+
image = load_image(url)
|
101 |
+
image = pidinet(
|
102 |
+
image, detect_resolution=1024, image_resolution=1024, apply_filter=True
|
103 |
+
)
|
104 |
+
```
|
105 |
+
<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_sketch.png"><img width="480" style="margin:0;padding:0;" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_sketch.png"/></a>
|
106 |
+
|
107 |
+
- Generation
|
108 |
+
```py
|
109 |
+
prompt = "a robot, mount fuji in the background, 4k photo, highly detailed"
|
110 |
+
negative_prompt = "extra digit, fewer digits, cropped, worst quality, low quality, glitch, deformed, mutated, ugly, disfigured"
|
111 |
+
|
112 |
+
gen_images = pipe(
|
113 |
+
prompt=prompt,
|
114 |
+
negative_prompt=negative_prompt,
|
115 |
+
image=image,
|
116 |
+
num_inference_steps=30,
|
117 |
+
adapter_conditioning_scale=0.9,
|
118 |
+
guidance_scale=7.5,
|
119 |
+
).images[0]
|
120 |
+
gen_images.save('out_sketch.png')
|
121 |
+
```
|
122 |
+
<a href="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/cond_sketch.png"><img width="480" style="margin:0;padding:0;" src="https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/res_sketch.png"/></a>
|
123 |
+
|
124 |
+
### Training
|
125 |
+
|
126 |
+
Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/t2i_adapter/README_sdxl.md).
|
127 |
+
|
128 |
+
The model is trained on 3M high-resolution image-text pairs from LAION-Aesthetics V2 with
|
129 |
+
|
130 |
+
- Training steps: 20000
|
131 |
+
- Batch size: Data parallel with a single gpu batch size of `16` for a total batch size of `256`.
|
132 |
+
- Learning rate: Constant learning rate of `1e-5`.
|
133 |
+
- Mixed precision: fp16
|
config.json
ADDED
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_class_name": "T2IAdapter",
|
3 |
+
"_diffusers_version": "0.21.0.dev0",
|
4 |
+
"_name_or_path": "valhalla/t2i-sketch",
|
5 |
+
"adapter_type": "full_adapter_xl",
|
6 |
+
"channels": [
|
7 |
+
320,
|
8 |
+
640,
|
9 |
+
1280,
|
10 |
+
1280
|
11 |
+
],
|
12 |
+
"downscale_factor": 16,
|
13 |
+
"in_channels": 3,
|
14 |
+
"num_res_blocks": 2
|
15 |
+
}
|
diffusion_pytorch_model.fp16.safetensors
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:9f88c53368d53bec6b70fc9d797ec12b88a512d2a7e9068d11fbf621917d9fdd
|
3 |
+
size 158060440
|
diffusion_pytorch_model.safetensors
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:95ffaafde9e0a13061205c1d701dd22326be75884b1adfc0c1a1626dfa72d0a4
|
3 |
+
size 316116778
|