I want to make a pseudo-GPT3(GPT2 with more layers), but it’s too large for my 2*V100. And when I try to train on CPU, it gives me an error:
Traceback (most recent call last):                  | 0/1260608 [00:00<?, ?it/s]

File “examples/language-modeling/train.py”, line 336, in 
main()
File “examples/language-modeling/train.py”, line 300, in main
trainer.train(model_path=model_path)
File “/home/ksjae/.local/lib/python3.7/site-packages/transformers/trainer.py”,
line 741, in train
tr_loss += self.training_step(model, inputs)
File “/home/ksjae/.local/lib/python3.7/site-packages/transformers/trainer.py”,
line 1055, in training_step
self.scaler.scale(loss).backward()
File “/home/ksjae/.local/lib/python3.7/site-packages/torch/cuda/amp/grad_scale
r.py”, line 156, in scale
assert outputs.is_cuda
AssertionError