How to parallel in GPU when finetuning
In my Finetune models , I wanna parallel my model, in multi-gpus,  my code is shown below: <SCODE>class FinetuneModel(nn.Module):
    def __init__(self, pretrained_model, ngpu = opt.gpuids):
        self.ngpu = ngpu
        super(FinetuneModel, self).__init__()
        self.features = pretrained_model
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512 * 4 * 4, 2048),
            ....           
))      
        
    def forward(self, x):
        gpuids = None
        if self.ngpu:
            gpuids = range(self.ngpu)
        features =  self.features(x)#self.features has already implemented data parallel
        return nn.parallel.data_parallel(self.classifier, features, device_ids=gpuids)

<ECODE> as far as I know , when doing <SCODE>features =  self.features(x)#self.features.forward has already implemented data parallel
score = nn.parallel.data_parallel(self.classifier, features, device_ids = gpuids)
<ECODE> <SCODE>score = nn.parallel.data_parallel([self.features,self.classifier], features, device_ids = gpuids)
<ECODE> which only does one broadcast
chenyuntc
Cysu
chenyuntc
As AlexNet and VGG contain lots of parameters in the FC layers. Syncing params on these layers will have large overhead. It’s faster to compute the FC layers only on one GPU.
Cysu
I see. great thanks.
chenyuntc
apaszke