Hm… I see. I ideally wanted to train some iterations with a part of my network frozen, and some iterations after that with it not frozen, and alternate.
I have 4 sub-networks(a,b,c,d). And there’s a big wrapper network (say N) that contains those 4 sub-networks, where the wrapper network’s flow goes  (a,b separately)->combined into c->then d. And I wanted to freeze subnetwork a’s weights (load pickled trained weights for subnetwork a and don’t train them).

Not considering alternate freezing/unfreezing, what’s the best way to achieve freezing weights in this case?
Would it be like the following? But Once DataParallel wraps the model, the “a” subnetwork is no longer accessible from model. What’s the best way of freezing subnetwork a’s weights for multi-GPU setting? Some way of maybe model.modules()? or model._modules[‘module’].a.parameters() ?

model = N()
model = nn.DataParallel(model)  
for parameter in model.a.parameters():
    parameter.requires_grad = False


If I wanted to alternate freezing/unfreezing, what’s the ideal way?