File size: 3,570 Bytes
0e1ab49 175b6b2 57bb9de 175b6b2 57bb9de 0e1ab49 175b6b2 a2d3b18 175b6b2 a2d3b18 175b6b2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
---
language:
- en
license: apache-2.0
library_name: peft
tags:
- music
datasets:
- ThatOneShortGuy/SongLyrics
base_model: OpenAssistant/falcon-7b-sft-mix-2000
---
# Musical Falcon
[OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000) model fine tuned using PEFT on
[Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics) to write lyrics to songs.
## Model Details
- **Finetuned from**: [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000)
- **Model Type**: Causal decoder-only transformer language model
- **Language**: English (and limited capabilities in German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian, Czech, Swedish);
- **License**: Apache 2.0
- **Contact**: Lol don't. This is just for fun.
## Usage
The basic basic format getting it in is:
```python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM
config = PeftConfig.from_pretrained("ThatOneShortGuy/MusicalFalcon")
model = AutoModelForCausalLM.from_pretrained("OpenAssistant/falcon-7b-sft-mix-2000")
model = PeftModel.from_pretrained(model, "ThatOneShortGuy/MusicalFalcon")
```
## Prompting
Considering this comes from [OpenAssistant/falcon-7b-sft-mix-2000](https://huggingface.co/OpenAssistant/falcon-7b-sft-mix-2000), it uses the same structure.
Two special tokens are used to mark the beginning of user and assistant turns: `<|prompter|>` and `<|assistant|>`. Each turn ends with a `<|endoftext|>` token.
The training prompt used the structure:
```
<|prompter|>Come up with the lyrics for a song from "{artist}" {"from " + year if year else ""} titled "{title}".<|endoftext|>
<|assistant|>Sure! Here are the lyrics:
{lyrics}
<|endoftext|>
```
However, it still seems to work just fine using:
```
<|prompter|>Write me a song titled "{title}"<|endoftext|><|assistant|>
```
or any anything of similar nature. Feel free to add a description of the song in there too.
## Examples
Prompt
```
<|prompter|>Write a song with the lyrics for a song from artist "Eminem" from 2018 titled "Suzy's Lemonade Stand"<|endoftext|>
<|assistant|>Sure! Here are the lyrics:
```
Result
```
[Intro]
I'm gonna make you famous, baby
[Verse 1]
It's been a long time since I had a lemonade stand
But I'm back in town and I'ma do it again
You know what? I've been on my own since I was born
So I'll be here all day long, so come on, let's go
I got some new tricks up my sleeve that I'd like to show
And I'mma get your name out of your mouth, so please don't blow
Let me see if I can get you to say my name
If not, then I'l just have to take it back to the drawing board
I gotta get this money, I's a little bit broke
I need some cash, I need some dough, I gotta get these clothes off
I wanna be a star, I wanna be the man
I want to be a superstar, I want to get my hands on them bands
```
## Training procedure
Trained on 1 epoch of all songs from [Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics) with "Rap" or "Hip" in the genre title.
- GPU: RTX Titan
- load_in_8bit: True
Further trained on ~8 epochs of the top 10% of [Song Lyrics](https://huggingface.co/datasets/ThatOneShortGuy/SongLyrics)
- GPU: RTX Titan
- load_in_8bit: False
- torch_dtype: torch.float16
- load_in_4bit: False
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: True
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: fp4
- bnb_4bit_use_double_quant: False
### Framework versions
- PEFT 0.5.0.dev0 |