In PyTorch, grads are set to None when the weights are initially initialized. Which means, if the grad of your weights is not None, then backpropagation has flowed through it (at least once). You noted that: list(TrainNet.model.parameters())[0].grad is not None --> True, which means backpropagation is doing something.
Before optimizer.zero_grad(), try printing out the grad of each weight and see what their values are. It’s difficult for me to follow all the operations you’re performing in your forward pass, but looking at your grads directly may provide some intuition on what’s going wrong. The key thing I would look for is if your grads are mostly zero or sparse. Feel free to post anything you find and have questions on. Additionally, the full update procedure would be good to post too. I’m wondering how selected_action_values are being computed.