Solved by David_Harvey in post #6 
              
              
                To my understanding, you should make it this way 
...
tmp_adv_ex = []
for idx, item in enumerate(adv_ex):
    tmp_adv_ex.append(adv_ex[idx] + purturbation)
adv_ex = torch.cat(tmp_adv_ex, dim=0) # now it won't cause in-place modifiy
...