I’m using an Nvidia RTX 3090 that has 24GB of dedicated VRAM. For some reason, I am still running out of memory… am I doing something wrong?
from transformers import GPTJForCausalLM, AutoTokenizer
import torch    

model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", torch_dtype=torch.float16, low_cpu_mem_usage=True).to("cuda")

Traceback (most recent call last):
  File "C:\Users\wolfg\code\aibrush-2\worker\gptjtest.py", line 5, in <module>
    model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", torch_dtype=torch.float16, low_cpu_mem_usage=True).to("cuda")
  File "C:\Users\wolfg\anaconda3\envs\vqgan\lib\site-packages\torch\nn\modules\module.py", line 852, in to
    return self._apply(convert)
  File "C:\Users\wolfg\anaconda3\envs\vqgan\lib\site-packages\torch\nn\modules\module.py", line 530, in _apply
    module._apply(fn)
  File "C:\Users\wolfg\anaconda3\envs\vqgan\lib\site-packages\torch\nn\modules\module.py", line 552, in _apply
    param_applied = fn(param)
  File "C:\Users\wolfg\anaconda3\envs\vqgan\lib\site-packages\torch\nn\modules\module.py", line 850, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 788.00 MiB (GPU 0; 24.00 GiB total capacity; 21.88 GiB already allocated; 399.81 MiB free; 21.89 GiB reserved in total by PyTorch)