Oh I see now. I didn’t realize changing the list to a torch tensor with torch.tensor removes it from the gradient history. Instead of using
currCriticVals = torch.tensor(currCriticVals[0:advantages.shape[0]], dtype=torch.float, requires_grad=True, device=device)
I used
currCriticValss = torch.cat(currCriticVals[0:advantages.shape[0]])
Where currCriticVals is a list of tensors.
Now that I changed it, the gradients propagate through the network. Looking back at torch.tensor vs. torch.cat makes perfect sense as to why it wasn’t working before.
Thank you for your help with this issue!