Hi, all!
I came across a problem when using multi-gpu training. Specifically, I’d like to load the supernet to CPU memory and optimize one of the branch on GPU in each iter (because the whole supernet is toooo big, which would cost a lot of GPU Memory if load all the parameters and pull down the power of GPU). But it’s hard for nn.DataParallel to pass parameters when using ‘‘model=nn.DataParallel(model, device_ids=engine.devices)’’ and ‘‘model.to(‘cpu’)’’  (because parameters can not broadcast when the model is allocated to cpu). I’m wondering if you have some better implementations?
Thanks a lot in advance.