config.json "max_position_embeddings": 40960,
Is this correct? I noticed this in the original model too. Looks like they added a 0 to the end of 4096. The actual model length should be 32768 I think?
"The default max_position_embeddings in config.json is set to 40,960. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance."
"The default max_position_embeddings in config.json is set to 40,960. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance."
correct!