Hi, recently I used the DDPG agent based on PyTorch to build a project.
When I was trying to read a batch of combinations of (state, action, reward, next_state) from the buffer and to update the weights of the 4 networks, it seemed loss.backward() function didn’t update the grads at all! There is just no data shown on the tensorboard page.
Here is the code for “update based on the buffer” part:
def learn(self, adj):
    if self.buffer.mem_cntr < self.batch_size:
        return
    state_vectors, action_vectors, rewards, next_state_vectors = \
                               self.buffer.sample_buffer(self.batch_size)

    adj = T.tensor(adj, dtype = T.float32).to(self.actor.device)
    for i in range(self.batch_size):
        state_vector = T.tensor(state_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        next_state_vector = T.tensor(next_state_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        action_vector = T.tensor(action_vectors[i], dtype = T.float32).unsqueeze(0).to(self.actor.device)
        reward = T.tensor(rewards[i], dtype = T.float32).to(self.actor.device)

        next_target_action_vector = self.target_actor.forward(next_state_vector, adj)
        next_target_q = self.target_critic(next_state_vector, next_target_action_vector, adj)
        q = self.critic.forward(state_vector, action_vector, adj)

        target_q = reward + self.gamma * next_target_q
    
        self.critic.optimizer.zero_grad()
        critic_loss = F.mse_loss(q, target_q)
        critic_loss.backward()
        self.critic.optimizer.step()
            
        self.actor.optimizer.zero_grad()
        actor_loss = -self.critic.forward(state_vector, 
                                              self.actor.forward(state_vector, adj), 
                                              adj) / self.batch_size
        actor_loss.backward()
        self.actor.optimizer.step()

    self.soft_update()

Screenshot from 2020-09-16 16-04-20685×742 123 KB
How to modify the codes to make the actor-network work correctly?