Vezora commited on
Commit
36951fa
1 Parent(s): aa82324

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -6
README.md CHANGED
@@ -11,13 +11,13 @@ license: apache-2.0
11
  **Creator** [Nicolas Mejia-Petit](https://twitter.com/mejia_petit)
12
 
13
  ### Overview
14
- Just one day after the release of **Mixtral-8x-22b**, we are excited to introduce our handcrafted experimental model, **Mistral-22b-V.02**. This model is a culmination of equal knowledge distilled from all experts into a single, dense 22b model. This model is not a single trained expert, rather its a compressed MOE model, turning it into a dense 22b mode. This is the first working MOE to Dense model conversion.
15
 
16
  ### Capabilities
17
  - **Math Proficiency**: The model exhibits strong mathematical abilities. Dispite not being trained on math.
18
  - **Better at Coding** The model is significantly better at coding, than V1, it passed some of my simple coding test, such as "Create a simple HTML site with a button that changes the background color to a random color", which V1 failed.
19
  - **More Cohesive** This V2 model is significantly more cohesive, and better at aunderstanding the prompts and answering with the appopriate answer.
20
- - **Highly Uncencored** Since this model was also Re-Alligned to be uncencored, it can just answer anything you ask.
21
  - **Multi Turn** The dataset this model trained on was mostly all multi turn conversations, spanning many different topics, with some emphasis on coding.
22
  - **Json Mode** I did train this model on answering in JSON and using JSON tools., I have yet to try it, in depth but preliminary test shows it works, including.
23
  - **Agent abilities** I did train this model on agent datasets, that teach it to do real world tasks such as picking up an object, and even navigating a webpage based off HTML.
@@ -28,10 +28,9 @@ Just one day after the release of **Mixtral-8x-22b**, we are excited to introduc
28
  Please note that Mistral-22b is still in a WIP. V.3 has started training now, with a different method than used before, this is to hopefully make the model more round in its internel knowlledge. Through my testing i found V2 to be a significant improvement over V.2, but I would like to do my testing with other methods.
29
 
30
  ### Upcoming Release: V.3
31
- V-3 will feature a different base model for testing purposed, however this model is pretty darn good for a second test. :)
 
32
 
33
- ### Background
34
- The decision to release this experimental version was prompted by someone attempting to replicate my experiment based on my tweets. We wanted to ensure our community has access to the official version first.
35
 
36
  ### Stay Updated
37
  **V.3**, coming soon! And is currently training, will be done in the next ~24 hours. 🌟Paper Coming Soon🌟
@@ -40,7 +39,7 @@ The decision to release this experimental version was prompted by someone attemp
40
 
41
  ### Usage:
42
  - This model requires a specific chat template, as the training format was Guanaco this is what it looks like:
43
- - "### System: You are a helpful assistant. ### Human###: Give me the best chili recipe you can ###Assistant:"
44
 
45
 
46
  ## Thank you!
 
11
  **Creator** [Nicolas Mejia-Petit](https://twitter.com/mejia_petit)
12
 
13
  ### Overview
14
+ Just two days after the release our release of **Mistral-22b-v0.1**, we are excited to introduce our handcrafted experimental model, **Mistral-22b-V.02**. This model is a culmination of equal knowledge distilled from all experts into a single, dense 22b model. This model is not a single trained expert, rather its a compressed MOE model, turning it into a dense 22b mode. This is the first working MOE to Dense model conversion.
15
 
16
  ### Capabilities
17
  - **Math Proficiency**: The model exhibits strong mathematical abilities. Dispite not being trained on math.
18
  - **Better at Coding** The model is significantly better at coding, than V1, it passed some of my simple coding test, such as "Create a simple HTML site with a button that changes the background color to a random color", which V1 failed.
19
  - **More Cohesive** This V2 model is significantly more cohesive, and better at aunderstanding the prompts and answering with the appopriate answer.
20
+ - **Highly Uncencored** Since this model was also Re-Alligned to be uncencored, it can just answer anything you ask. So use at your own risk, we take no responsibility for your generated responces.
21
  - **Multi Turn** The dataset this model trained on was mostly all multi turn conversations, spanning many different topics, with some emphasis on coding.
22
  - **Json Mode** I did train this model on answering in JSON and using JSON tools., I have yet to try it, in depth but preliminary test shows it works, including.
23
  - **Agent abilities** I did train this model on agent datasets, that teach it to do real world tasks such as picking up an object, and even navigating a webpage based off HTML.
 
28
  Please note that Mistral-22b is still in a WIP. V.3 has started training now, with a different method than used before, this is to hopefully make the model more round in its internel knowlledge. Through my testing i found V2 to be a significant improvement over V.2, but I would like to do my testing with other methods.
29
 
30
  ### Upcoming Release: V.3
31
+ - V-3 will feature a different base model for testing purposed, however this model is pretty darn good for a second test. :)
32
+ - I have done some preliminary results with my different base model, and it appears to achieve a lower loss after the first epoch compared to the other base model used for v0.1 and v0.2. so we have started training v0.3 with the new base model and with the longer dataset.
33
 
 
 
34
 
35
  ### Stay Updated
36
  **V.3**, coming soon! And is currently training, will be done in the next ~24 hours. 🌟Paper Coming Soon🌟
 
39
 
40
  ### Usage:
41
  - This model requires a specific chat template, as the training format was Guanaco this is what it looks like:
42
+ - "### System: You are a helpful assistant. ### Human###: Give me the best chili recipe you can ###Assistant: Here is the best chili recipe..."
43
 
44
 
45
  ## Thank you!