As per your suggestion, I have included torch.autograd.set_detect_anomaly(True) in my code. Following are the details of the error,
RuntimeError                              Traceback (most recent call last)
<ipython-input-69-599bd4b36306> in <module>()
     29         # Backward and optimize
     30         optimizer.zero_grad()
---> 31         loss.backward()
     32         optimizer.step()
     33 

1 frames
/usr/local/lib/python3.7/dist-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
    305                 create_graph=create_graph,
    306                 inputs=inputs)
--> 307         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
    308 
    309     def register_hook(self, hook):

/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    154     Variable._execution_engine.run_backward(
    155         tensors, grad_tensors_, retain_graph, create_graph, inputs,
--> 156         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
    157 
    158 

RuntimeError: Function 'ExpBackward0' returned nan values in its 0th output.