Hi,
I’m trying to get intermedia output using forward hooks. Howerver with multiple-gpus, this does not work. Each gpu will receive a fraction of the input, so we need to aggregate the results coming from different gpus. I find data_parallel meet my need and I write a simple test program below. But weird things happened.
I give 3 inputs to vgg model in turn. The first is all -1 and the second is all zeros and the third is all 1.

For the first input, the output is None(I test, whaterer the first input is, it outputs None without any change)
For the second input, the output is not None but not all zeros(In fact when you feed all zeros tensor to net with ReLU, the intermedia is all zeros naturally). However, look closely, the output is corresponding to the first input
For the third input, the output corresponds to the second input, are all zeros.

Am I missing something important and  writing problematic code  ? Please let me known.
import torch
from torchvision.models.vgg import vgg19

class Wrapper(torch.nn.Module):
    def __init__(self, model):
        super(Wrapper, self).__init__()
        self.model = model
        self.target_outputs = None

        def forward_hook(_, __, output):
            self.target_outputs = output.detach()
        self.model.features[2].register_forward_hook(forward_hook)

    def forward(self,input):
        self.model(input)
        return self.target_outputs


model = vgg19()
model = model.cuda(4)
wrapper = Wrapper(model)

devices = [4, 5]

input1 = torch.randn(60,3,224,224).fill_(-1).cuda(4)
out1 = torch.nn.parallel.data_parallel(wrapper, input1, devices)
print(out1)
input2 = torch.randn(60,3,224,224).fill_(0).cuda(4)
out2 = torch.nn.parallel.data_parallel(wrapper, input2, devices)
print(out2.shape)
input3 = torch.randn(60,3,224,224).fill_(1).cuda(4)
out3 = torch.nn.parallel.data_parallel(wrapper, input3, devices)
print(out3.shape)


output:
None
(60, 64, 224, 224)
(60, 64, 224, 224)