That must be some very specific interaction in your code then 
Have you tried .clone() on each input and output of the CrossEntropyLoss? If one of them fixes the issue, then it is this one that was modified inplace   like loss_s = reader_loss(log_probs_S_per_ex.unsqueeze(0).clone(), gt_start.clone()).clone()