tangledgroup
/

tangled-alpha-0.10-core

Text Generation

Model card Files Files and versions Community

mtasic85 commited on Mar 19

Commit

bf6e2b3

·

1 Parent(s): c60dfe3

pretrain core

Files changed (1) hide show

config-0.json +1 -1

config-0.json CHANGED Viewed

@@ -14,7 +14,7 @@
   "max_position_embeddings": 131072,
   "mlp_bias": false,
   "model_type": "llama",
-  "num_attention_heads": 32,
   "num_hidden_layers": 32,
   "num_key_value_heads": 8,
   "pretraining_tp": 1,

   "max_position_embeddings": 131072,
   "mlp_bias": false,
   "model_type": "llama",
+  "num_attention_heads": 16,
   "num_hidden_layers": 32,
   "num_key_value_heads": 8,
   "pretraining_tp": 1,