paulilioaica
/

PhiMiX-2x2B

Text Generation

Mixture of Experts

cognitivecomputations/dolphin-2_6-phi-2

rhysjones/phi-2-orange

Inference Endpoints

Model card Files Files and versions Community

paulilioaica commited on Jan 29

Commit

eda3d62

•

1 Parent(s): dda40f2

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -24,7 +24,7 @@
   "n_layer": 32,
   "n_positions": 2048,
   "num_experts_per_tok": 2,
-  "num_local_experts": 4,
   "resid_pdrop": 0.1,
   "rotary_dim": 32,
   "tie_word_embeddings": false,

   "n_layer": 32,
   "n_positions": 2048,
   "num_experts_per_tok": 2,
+  "num_local_experts": 2,
   "resid_pdrop": 0.1,
   "rotary_dim": 32,
   "tie_word_embeddings": false,