I’ve looked at many articles and have been Googling for a few days now without being able to fix the issue I’m having. I am having an issue with the autograd PyTorch function in my sequential models. I am trying to implement the PPO algorithm, but for some reason, the gradients don’t propagate to my networks after calling backward on the loss term as seen below in my critic network

Capture1489×606 103 KB

In fact, the gradients propagate all the way to the vector of outputs from the network, but they don’t actually propagate through the network (currCriticVals is an array storing values directly from the critic network):
  
      

      ImgBB
  

  
    

Capture2 1

  Image Capture2 hosted in ImgBB


  

  
    
    
  

  


(Sorry for the link. As a new user, I can only have one embed and two links)
Also, note that I am running this on my CPU at the moment if that may cause any problems.
The full code can be found at the following GitHub link:
  
      

      GitHub
  

  
    

GitHub - gmongaras/PPO_CartPole: Using PPO, I am attempting to solve the...

  Using PPO, I am attempting to solve the cartpole environment - GitHub - gmongaras/PPO_CartPole: Using PPO, I am attempting to solve the cartpole environment


  

  
    
    
  

  


I have a few debugging lines near the backward function in case you realize it looks a bit messy.
Any help would be greatly appreciated! Thank you!