Oobabooga Chat Errors
#9
by
PieDev
- opened
Oobabooga seems to be unable to run the model properly as shown below:
I'm getting the same issue as of this post. Been playing around with different settings, but haven't been able to improve things much.
Been running this on runpod with the latest image, on a single L40
Edit note:
Solved the problem by ensure these instructions were properly followed:To use the increased context, set the Loader to ExLlama, set max_seq_len to 8192 or 4096, and set compress_pos_emb to 4 for 8192 context, or to 2 for 4096 context.