BlinkDL commited on
Commit
0be695e
1 Parent(s): 060b78b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -0
README.md CHANGED
@@ -1,3 +1,27 @@
1
  ---
 
 
 
 
 
 
 
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - pytorch
6
+ - text-generation
7
+ - causal-lm
8
+ - rwkv
9
  license: apache-2.0
10
+ datasets:
11
+ - The Pile
12
+
13
  ---
14
+
15
+ # RWKV-4 7B
16
+
17
+ ## Model Description
18
+
19
+ RWKV-4 7B is a L32-D4096 causal language model trained on the Pile. See https://github.com/BlinkDL/RWKV-LM for details.
20
+
21
+ ** Note: It's a BF16 model, and it may overflow if you are using FP16 (probably fixable by rescaling the weights). **
22
+
23
+ At this moment you have to use my Github code (https://github.com/BlinkDL/RWKV-LM) to run it.
24
+
25
+ ctx_len = 1024
26
+ n_layer = 32
27
+ n_embd = 4096