EDIT: You were right, this works. I had a different bug preventing the proper usage of the random number generator. Thanks for your help!
My current workaround was to pass pregenerated random tensors as generator objects to the modules manually. For some reason that does not result in the same output for me. E.g. in the above outlined logic I do,
import environment

torch.manual_seed(2019)
noise_generator = (t for t in torch.randn(10))
world = environment.World(noise_generator)

for t in time:
    observation = world.step(dt)
    model.update(observation)

but the observations are different for different models and I am not sure why that is the case.