SmallDoge
/

Transformer-MHA-1700M-16K

Text Generation

Model card Files Files and versions

JingzeShi commited on Jun 11

Commit

65a8232

·

verified ·

1 Parent(s): 432315d

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -18,7 +18,7 @@
   "intermediate_size": 6144,
   "is_moe": false,
   "keep_window_size": 8192,
-  "max_position_embeddings": 8192,
   "model_type": "doge2",
   "norm_topk_prob": false,
   "num_attention_heads": 16,

   "intermediate_size": 6144,
   "is_moe": false,
   "keep_window_size": 8192,
+  "max_position_embeddings": 16384,
   "model_type": "doge2",
   "norm_topk_prob": false,
   "num_attention_heads": 16,