Interestingly, cloning gt really works!
loss_s = reader_loss(log_probs_S_per_ex.unsqueeze(0), gt_start.clone())
loss_e = reader_loss(log_probs_E_per_ex.unsqueeze(0), gt_end.clone())

and after closer investigation, I have found the guilty! It is the += operator applied to offset
Changing
offset = log_probs_S_per_ex[:flat_gt_index[example_ix]].numel()
# add offset to closed-domain ground truths
gt_start += offset
gt_end += offset

to
offset = log_probs_S_per_ex[:flat_gt_index[example_ix]].numel()
# add offset to closed-domain ground truths
gt_start = gt_start+ offset
gt_end = gt_end+ offset

solves this issue (without any explicit cloning)!