try doing this segment on cpu, the issue is that some operators used during backward are non-deterministic, so gradients differ in re-runs; in particular, indexing/gathering with many-to-one element correspondence