|
--- |
|
license: creativeml-openrail-m |
|
library_name: diffusers |
|
tags: |
|
- stable-diffusion |
|
- stable-diffusion-diffusers |
|
- text-to-image |
|
- diffusers |
|
- lora |
|
inference: true |
|
base_model: CompVis/stable-diffusion-v1-4 |
|
datasets: |
|
- sylvainlapeyrade/kanji_english_meaning |
|
language: |
|
- en |
|
pipeline_tag: text-to-image |
|
--- |
|
|
|
# LoRA text2image fine-tuning - sylvainlapeyrade/kanji2english |
|
<i>This is a model ran on only one epoch on Google Colab free version for proof of concepts purpose. Do not expect extraordinary resutls.</i> |
|
|
|
<img src="./generated_kanji_examples.jpg" width="500"> |
|
|
|
These are LoRA adaptation weights for CompVis/stable-diffusion-v1-4. The weights were fine-tuned on the sylvainlapeyrade/kanji_english_meaning dataset to generate Kanji images based on English descriptions. This model provides a novel approach to visualizing the artistic representation of Kanji characters through a text-to-image generation process, using the powerful Stable Diffusion architecture enhanced with LoRA layers for fine-tuning. |
|
|
|
## Model Description |
|
|
|
This model uses LoRA (Low-Rank Adaptation) layers to fine-tune the Stable Diffusion v1-4 model specifically for the task of generating Kanji images from English text descriptions. The fine-tuning process was conducted using the sylvainlapeyrade/kanji_english_meaning dataset, which contains a collection of Kanji characters and their corresponding English meanings. |
|
|
|
## How to Use |
|
|
|
This model is intended to be used with the `diffusers` library. Here is an example of how to generate an image from text: |
|
|
|
```python |
|
from diffusers import LoraDiffusionPipeline |
|
|
|
# Load the pipeline |
|
pipe = LoraDiffusionPipeline.from_pretrained("sylvainlapeyrade/kanji2english") |
|
|
|
# Generate an image |
|
prompt = "a kanji meaning a Doge" |
|
image = pipe(prompt).images[0] |
|
|
|
# Save or display the image |
|
image.save("kanji_doge.png") |
|
``` |
|
|
|
## Limitations and Bias |
|
|
|
This model has been trained specifically for generating Kanji characters from English text inputs with the <i>"a kanji meaning"</i> sequence and may not generalize well to other text-to-image tasks outside this scope. Additionally, the accuracy of the generated images may vary depending on the complexity and rarity of the Kanji. |
|
|
|
## Acknowledgements |
|
|
|
Special thanks to the creators of the KANJIDIC and KanjiVG projects for providing the data sources used during training. |
|
|
|
## License |
|
|
|
This model is released under the CreativeML Open RAIL-M license, which allows for non-commercial use, sharing, and adaptation with attribution. |