turboderp

/

Mistral-Large-Instruct-2411-exl3

like 4

Hey so taking a brief look at the ppl v bpw chart for llama 70B showed that the most dramatic difference between exl2 and exl3 is at the lower end with bpw < 4. While significantly more efficient overall, I'm very curious to see how much of an improvement it is for Mistral large in particular because the current sweetspot is 2.7-2.85 bpw for 2x 3090s (depending on context) and where the disparity is most likely to be felt when updating.

Any chance you would be able to do a similar chart for 123B? It's a bit of work but I think it presents a very exciting opportunity to possibly 'upgrade' from 70B models to 123b models by default depending on the results.