witl:

This is more related to your experience in the field.


Unfortunately, I’m not experienced with RL and cannot be of much help here. 
However, for the first question regarding the creation of one-hot encoded tensors:
I don’t know which shape tensor and one_hot should have in your example exactly, but you could try to use F.one_hot or one_hot.scatter to create a one-hot encoded tensor, which might be faster than a loop.