Hmm then it is really weird because broadcast on dense contiguous tensors don’t copy on the current device as expected. Could you double check that OOM happens when broadcasting the modules?, i.e. in replicate(...) of replicate.py