While searching for a solution I cam upon this thread: https://github.com/pytorch/pytorch/issues/3358 2, suggesting to add the below code under loss.backwards():
loss.backward()

for param in net.parameters():
    print(param.grad.data.sum())

unless I understood it wrong, this is to check if the computed gradients are returning any non-zero values. If they are non-zero then the learning rate is too small.
But the time when I get non-zero gradients is when the network is showing that its learning, but at zeros values I get a loss of 1.0.
tensor(-0.0001, device='cuda:0')
tensor(3.8132e-09, device='cuda:0')
tensor(1.5014e-06, device='cuda:0')
tensor(-0.0006, device='cuda:0')
tensor(1.4931e-09, device='cuda:0')
tensor(2.0285e-06, device='cuda:0')
tensor(-0.0003, device='cuda:0')
tensor(3.4502e-10, device='cuda:0')
tensor(1.4271e-06, device='cuda:0')
tensor(-0.0005, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(1.6241e-06, device='cuda:0')
tensor(-0.0003, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(1.2744e-06, device='cuda:0')
tensor(-0.0003, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(1.5523e-06, device='cuda:0')
tensor(-0.0006, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(2.0451e-06, device='cuda:0')
tensor(-0.0009, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(2.3836e-06, device='cuda:0')
tensor(-0.0011, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(2.6307e-06, device='cuda:0')
tensor(-0.0038, device='cuda:0')
tensor(6.4067e-10, device='cuda:0')
tensor(6.6819e-06, device='cuda:0')
tensor(-0.0074, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(2.7682e-06, device='cuda:0')
tensor(0.0008, device='cuda:0')
tensor(-0.0001, device='cuda:0')
tensor(-0.0000, device='cuda:0')
tensor(-0.0044, device='cuda:0')
tensor(-0.0001, device='cuda:0')
tensor(-0.0052, device='cuda:0')
tensor(-0.0007, device='cuda:0')
tensor(-0.0053, device='cuda:0')
tensor(-0.0004, device='cuda:0')
tensor(-0.0008, device='cuda:0')
tensor(-0.0021, device='cuda:0')
Iteration:  7  - idx:  6 of  297 - Training Loss:  0.83579021692276 - Training Error:  0.90537
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
Iteration:  8  - idx:  7 of  297 - Training Loss:  1.0 - Training Error:  1.0

Sorry for the very beginner question but the github thread suggests that zero gradient values are preferred, but the time I get high loss is when my gradients are zero.
I also tried lowering my learning rate from 1e-2 to 1e-3 but that did not improve or fix anything.
What am I doing wrong?
Thank you