GOT CUDA out of memory

by weiminw - opened Dec 1, 2023

Dec 1, 2023

when I run the 4bit model in vllm. I got following error. I am using rtx4090 (24G GPU memory).

01-ai org Dec 2, 2023

Please make sure no other processors occupying the GPU memory.
24G memory is enough to run 34B-4bits version

tianjun changed discussion status to closed Dec 2, 2023

Yhyu13

Dec 2, 2023

@weiminw You might want to manully set ctx len to 4096

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment