Gibberish output on 4bpw and multiGPU system
#2
by
Panchovix
- opened
Hello there, thanks for the quants!
I have been trying the 4bpw one but I get just gibberish and/or nonsense output.
I'm running it on Fedora, 208GB VRAM alongside 7 GPUs, exllamav3 dev and tabbyapi.
The issue and the fix ended up being completely different from what I proposed, but the dev branch should be fixed now.
MikeRoz
changed discussion status to
closed