damerajee
/

hathi-moe-test

Text Generation

Mixture of Experts

damerajee/Gaja-v1.00

ai4bharat/Airavata

Model card Files Files and versions Community

damerajee commited on Mar 29

Commit

6bbe7f1

•

1 Parent(s): 6d88f34

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -18,7 +18,7 @@
   "initializer_range": 0.02,
   "intermediate_size": 24576,
   "max_position_embeddings": 8192,
-  "model_type": "gemmoe",
   "num_attention_heads": 16,
   "num_experts_per_tok": 2,
   "num_hidden_layers": 28,

   "initializer_range": 0.02,
   "intermediate_size": 24576,
   "max_position_embeddings": 8192,
+  "model_type": "hixtral",
   "num_attention_heads": 16,
   "num_experts_per_tok": 2,
   "num_hidden_layers": 28,