gbenson
/

dom-tokenizer-100k

Model card Files Files and versions Community

gbenson commited on May 17, 2024

Commit

391319b

·

1 Parent(s): 80bd862

Increase model_max_length

Files changed (1) hide show

tokenizer_config.json +1 -1

tokenizer_config.json CHANGED Viewed

@@ -112,7 +112,7 @@
   "do_lower_case": false,
   "eos_token": "[EOS]",
   "mask_token": "[MASK]",
-  "model_max_length": 512,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",
   "strip_accents": null,

   "do_lower_case": false,
   "eos_token": "[EOS]",
   "mask_token": "[MASK]",
+  "model_max_length": 134217728,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",
   "strip_accents": null,