Well I printed the torch.sum(param.grad ** 2).item() for all the params as print(param_name,torch.sum(param.grad ** 2).item()) and found that there were nonzero results upto the action_head of Policy. That means that there were some problem at self.layer_out = nn.Sequential(nn.Linear(32, 1), nn.Softmax(dim=-1)) of the Ploicy. I changed it to self.layer_out = nn.Sequential(nn.Linear(32, 1)) and the problem seems to be solved. But can anyone tell me why nn.Softmax(dim=-1) was blocking the backprop to propagate further?