Something just occured to me. If we want PyTorch to be able to compute the gradients, that means that the whole computation, including action selection has to be done within the framework. I mean that in this case, you can’t compute the policy distribution, convert it to numpy, make your stochastic pick, use it, store it and later on, when the episode is over use it back, right?
Because then the computation graph would have holes and elements missing, am I right?
Thanks !