Would there by bnb version for the UD quants?
#5
by
Sandbo
- opened
This is a great size to run for a medium-scale GPU system for example with 4xA6000 Ada. I am using vLLM but GUFF isn't well supported.
I see that some other models are provided with bnb versions, would you considering adding it for the 235B 2507?
I'm also really hoping for BNB versions of the 235B 2507 model. I'm working with 4×H100s in a research cloud environment and would love to be able to fine-tune the 2507 version using Unsloth.
Hi guys thank you for your interest we really appreciate it. We will see what we can do and update you guys! :)