Thank you for your reply! I’m hoping to update both the Actor and Discriminator networks in terms of the L_Final loss function where the Actor network is updated through the L_CLIP loss function and the Critic network is updated through the L_VF function. Both L_CLIP and L_VF are added together to make L_Final which is what .backward() is called on. S is just some noise added on to the loss which shouldn’t affect the backward pass and is why I set it to “requires_grad=False”
L_CLIP = torch.min(r_ts*advantages, torch.clip(r_ts, 1-epsilon, 1+epsilon)*advantages).mean()

# Calculate the critic loss (L_VF)
L_VF = torch.pow(rewards-currCriticVals, 2).mean()

# Get the entropy bonus from a normal distribution
S = torch.tensor(np.random.normal(), dtype=torch.float, requires_grad=False)
 
#Calculate the final loss (L_CLIP_VF_S)
L_Final = L_CLIP - self.c1*L_VF + self.c2*S

What I’m hoping is that the Critic and Actor networks get a gradient for all parameters. In a simple case, L_VF is the loss for the critic and is what I’m mostly paying attention to when debugging. L_VF is composed of the values received from the critic. So, the process to calculate L_VF is as follows:

collect n critic values and put them in an array
currCriticVal = self.critic(observation) # Get value from critic
self.storeMemory(...,currCriticVal,...) # Store value in list

turn the array into a tensor and remove the last value:
currCriticVals = torch.tensor(currCriticVals[0:advantages.shape[0]], dtype=torch.float, requires_grad=True, device=device)

Calculate the L_VF loss for the critic network
L_VF = torch.pow(rewards-currCriticVals, 2).mean()


The interesting part is that currCriticVals receives a grad, but the Critic parameters do not and this is the problem I am having.
I reviewed my code again, I didn’t notice any of the following potential causes you mentioned. I used numpy once, but that was for input into the Critic network, and that value shouldn’t need a gradient. Before calculating the loss, I did rewrap a few tensors, but changing the values from say x = torch.tensor(x) to x2 = torch.tensor(x) didn’t solve the gradient issue. I also used argmax once, but the value received from argmax didn’t need a gradient as it’s just input into one of the networks.