How to extend to 32K?
#3
by
brucethemoose
- opened
The model card says:
Both of them are trained with 4K sequence length and can be extended to 32K during inference time.
But... What does this mean? The max size appeaes to be 4K:
"max_position_embeddings": 4096
By extend to 32K, do you mean RoPE alpha scaling? Something else?
brucethemoose
changed discussion title from
Extend to 32K?
to Hwo to extend to 32K?
brucethemoose
changed discussion title from
Hwo to extend to 32K?
to How to extend to 32K?
The model card says:
Both of them are trained with 4K sequence length and can be extended to 32K during inference time.
But... What does this mean? The max size appeaes to be 4K:
"max_position_embeddings": 4096
By extend to 32K, do you mean RoPE alpha scaling? Something else?
You can try to set "max_position_embeddings": 32768 directly.
Thanks!
brucethemoose
changed discussion status to
closed