I think I found the problem: If I cast the numpy arrays img and seg as int64 directly there is no error occuring anymore.
NIFTI images can potentially have float values. torch.stack could not cast the inputs to the right datatype, giving an error.
With this code I can reproduce the error:
from torch import Tensor, stack, short, float, ones
import numpy as np

a = ones([2,2],dtype=short)
b = ones([2,2],dtype=float)
storage = a.storage()._new_shared(8)
out = a.new(storage)
stack([b,a], out=out)

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-111-eb9ed3493ddd> in <module>()
      6 storage = a.storage()._new_shared(8)
      7 out = a.new(storage)
----> 8 stack([b,a], out=out)

RuntimeError: result type Float can't be cast to the desired output type Short

while the following code gets no error:
from torch import Tensor, stack, short, float, ones
import numpy as np

a = ones([2,2],dtype=short)
b = ones([2,2],dtype=float)
stack([b,a])

The problem is the inplace stacking and predefinition to save memory which collate of the dataloader does.
So, in conclusion, if you are having similar issues check your datatypes.