Metrics for 110k context size?
#4
by
Thireus
- opened
First of all, thank you for providing these useful metrics. Would you be able to provide pp and eval t/s for 110k context size please?
You're welcome, glad you found it useful! I wish I had the hardware to run with 110k context size! :) Highest I can go 70k-90k token, but I have to choose slower settings to go past 75k, so the benchmak would not be comparing apples to apples.
FWIW the model perofrmance for coding starts to degrade a little past 32k and becomes very noticable past 64k tokens where some of my system instructions are no longer being followed. The performance degradation on larger context is a known problem for all the current models (even the ones with 1mil context don't performe as well past 64k tokens).