I just did a test out of curiosity. I am setting the gradient to 1/(sample_index + 1) for each sample before I call optimizer.step. The losses are now 100% identical when changing batch sizes and not the device. But changing the device still changes the loss.
This new observation lets me guess:

Differences between CPU and GPU are due to different hardware and probably rounding. Is that correct?
My original problem does not stem from the optimizers but from the way gradients are calculated. Since I am stacking my samples, I am assuming that the stacking operation has some effects on the gradients? Or maybe the loss function? What can I do about that?