Hi Liu, thank you for your reply! I know your point but for some reason, I have to replace some layers in the original actor and critic network, the networks consist of GCN(Graph Convolution Network) layers and linear layer, instead of mere Linear layers now. For GCN layers, I have two inputs: state_vector, which is a 1XN tensor, and an adjacency matrix, which is an NXN tensor. Inside the layer, the following computation is performed:
[weight matrix]     X      [input]        X         [adjacency matrix]
(out_dim X in_dim)  X    (in_dim X N)     X               (N X N)

and the output is a (out_dim X N) tensor, then this output is passed to the next GCN layer
As you can see, the first input(state_vector) is an 1 X N tensor, but the batch tensor is a (batch_size X N) tensor, which makes it impossible to pass it to the GCN layer, this is the reason I have to use a for loop to extract a single tuple of (state, action, reward, next_state) to make sure the dimensions match.
If I don’t update the weights in each loop, does that count as “simultaneously”?
To be honest, I am not sure if I have written the correct code to perform loss.backward() and optimizer.step()
Can I recode in this way?
self.critic.optimizer.zero_grad()
self.actor.optimizer_zero_grad()

# for loop to extract tuples and perform loss.backward()

self.critic.optimizer.step()
self.actor.optimizer.step()