rluukkon commited on
Commit
f938b23
1 Parent(s): dcd733d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +62 -1
README.md CHANGED
@@ -3,5 +3,66 @@ language:
3
  - fi
4
  pipeline_tag: text-generation
5
  ---
 
6
 
7
- GPT-3 xl for Finnish. Documentation coming soon!
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  - fi
4
  pipeline_tag: text-generation
5
  ---
6
+ Generative Pretrained Transformer with 1.5B parameteres for Finnish.
7
 
8
+ TurkuNLP Finnish GPT-3-models are a model family of pretrained monolingual GPT-style language models that are based on BLOOM-architecture.
9
+ Note that the models are pure language models, meaning that they are not [instruction finetuned](https://arxiv.org/abs/2203.02155) for dialogue
10
+ or answering questions.
11
+
12
+ These models are intended to be used as foundational models that can be e.g. instruction finetuned to serve as modern chat-models.
13
+
14
+
15
+
16
+ **Parameters**
17
+ | Model | Layers | Dim | Heads | Params |
18
+ |--------|--------|------|-------|--------|
19
+ | Small | 12 | 768 | 12 | 186M |
20
+ | Medium | 24 | 1024 | 16 | 437M |
21
+ | Large | 24 | 1536 | 16 | 881M |
22
+ | XL | 24 | 2064 | 24 | 1.5B |
23
+ | ”3B” | 32 | 2560 | 32 | 2.8B |
24
+ | ”8B” | 32 | 4096 | 32 | 7.5B |
25
+ | "13B" | 40 | 5120 | 40 | 13.3B |
26
+
27
+
28
+ **Datasets**
29
+
30
+ We used a combination of multiple Finnish resources.
31
+
32
+ * Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html
33
+ mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
34
+ * Common Crawl Finnish https://TODO
35
+ * Finnish Wikipedia https://fi.wikipedia.org/wiki
36
+ * Lönnrot Projekti Lönnrot http://www.lonnrot.net/
37
+ * ePub National library ”epub” collection
38
+ * National library ”lehdet” collection
39
+ * Suomi24 The Suomi 24 Corpus 2001-2020 http://urn.fi/urn:nbn:fi:lb-2021101527
40
+ * Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
41
+ * STT Finnish News Agency Archive 1992-2018 http://urn.fi/urn:nbn:fi:lb-2019041501
42
+ * Yle Finnish News Archive 2011-2018 http://urn.fi/urn:nbn:fi:lb-2017070501
43
+ * Yle Finnish News Archive 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050401
44
+ * Yle News Archive Easy-to-read Finnish 2011-2018 http://urn.fi/urn:nbn:fi:lb-2019050901
45
+ * Yle News Archive Easy-to-read Finnish 2019-2020 http://urn.fi/urn:nbn:fi:lb-2021050701
46
+ * ROOTS TODO
47
+
48
+
49
+ **Sampling ratios**
50
+
51
+ |Dataset | Chars | Ratio | Weight | W.Ratio |
52
+ |----------|--------|---------|--------|---------|
53
+ |Parsebank | 35.0B | 16.9\% | 1.5 | 22.7\%|
54
+ |mC4-Fi | 46.3B | 22.4\% | 1.0 | 20.0\%|
55
+ |CC-Fi | 79.6B | 38.5\% | 1.0 | 34.4\%|
56
+ |Fiwiki | 0.8B | 0.4\% | 3.0 | 1.0\%|
57
+ |Lönnrot | 0.8B | 0.4\% | 3.0 | 1.0\%|
58
+ |Yle | 1.6B | 0.8\% | 2.0 | 1.4\%|
59
+ |STT | 2.2B | 1.1\% | 2.0 | 1.9\%|
60
+ |ePub | 13.5B | 6.5\% | 1.0 | 5.8\%|
61
+ |Lehdet | 5.8B | 2.8\% | 1.0 | 2.5\%|
62
+ |Suomi24 | 20.6B | 9.9\% | 1.0 | 8.9\%|
63
+ |Reddit-Fi | 0.7B | 0.4\% | 1.0 | 0.3\%|
64
+ |**TOTAL** | **207.0B** | **100.0\%** | **N/A** | **100.0\%** |
65
+
66
+
67
+
68
+ More documentation and a paper coming soon.