Solved by tom in post #4 
              
              
                This is your setup doing it. I don’t think there is much that PyTorch can do for you here, because your operation is not inplace. 
When you do pars*sample that requires grad. In turn the cat produces a tensor that requires grad and refers to pars*sample and the previous out as inputs. Then leaveTopK…