Solved by CedricLy in post #4 
              
              
                OK, so the found problem is the tensor ls_fake. 
After doing optimizerD.step(), the gradients have to be recalculated, before using it for optimizer.step().