Okay I have a problem with the implementation of this now.
I do clone the action probabilities to avoid an inplace operation like this:
pActions, values = self.forward(observations)
legalActions = pActions.clone()
legalActions[1 - valids] = 0
actions = torch.multinomial(legalActions, 1).long()
logProbs = torch.log(torch.gather(pActions, -1, actions))

However now I get an cuda runtime error that seems to come from the multinomial operation. However the traceback says the error is triggered by the line where I calculate the policy loss with (negLogProbs * advt).mean() after which I call lossPolicy.backward():
/home/marcel/anaconda3/envs/pytorch/pytorch/aten/src/THC/THCTensorRandom.cuh:182: void sampleMultinomialOnce(long *, long, int, T *, T *, int, int) [with T = float, AccT = float]: block: [0,0,0], thread: [2,0,0] Assertion `THCNumerics<T>::ge(val, zero)` failed.
THCudaCheck FAIL file=/home/marcel/anaconda3/envs/pytorch/pytorch/aten/src/THC/generated/../THCReduceAll.cuh line=339 error=59 : device-side assert triggered
Traceback (most recent call last):
 ....

  File "/home/marcel/Projects/test/train.py", line 40, in train
    lossPolicy = (negLogProbs * advt).mean()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /home/marcel/anaconda3/envs/pytorch/pytorch/aten/src/THC/generated/../THCReduceAll.cuh:339


Is there anything wrong with the cloning of the legal actions? Does this backpropagate correctly?