Thank you for your answer.
Now it’s not throwing the memory error. But giving TypeError: Broadcast function not implemented for CPU tensors. I think the reason is because the model was never moved to CUDA memory
Traceback (most recent call last):
  File "hckr_rnk.py", line 173, in <module>
    train(model, device, train_loader, optimizer, 50,test_loader)
  File "hckr_rnk.py", line 105, in train
    output = model(data)
  File "/home/jmandivarapu1/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jmandivarapu1/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 122, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/home/jmandivarapu1/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 127, in replicate
    return replicate(module, device_ids)
  File "/home/jmandivarapu1/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 12, in replicate
    param_copies = Broadcast.apply(devices, *params)
  File "/home/jmandivarapu1/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 13, in forward
    raise TypeError('Broadcast function not implemented for CPU tensors')
TypeError: Broadcast function not implemented for CPU tensors