TheBloke

/

Llama-2-70B-Chat-GPTQ

like 260

I am running this model with two GPU. I get the following error:

OutOfMemoryError: CUDA out of memory. Tried to allocate 2.37 GiB (GPU 0; 23.69 GiB total capacity; 
19.71 GiB already allocated; 914.75 MiB free; 21.86 GiB reserved in total by PyTorch) 
If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF