It seems to work only when I wrap the forward call in with model.no_sync() and set the number of workers to 0. This makes the loading very slow however. And when I try to use a config with Deepspeed it throws an error that the model object has no attribute no_sync.