Thanks for the great solution.
For python3.7, we need to use threading.get_ident() because get_native_id() is only defined in python3.8.
The following code works for me.
In [4]: import torch
   ...: import threading
   ...: from collections import defaultdict
   ...: from torchvision.models.vgg import vgg19
   ...: 
   ...: class Wrapper(torch.nn.Module):
   ...:     def __init__(self, model):
   ...:         super(Wrapper, self).__init__()
   ...:         self.model = model
   ...:         self.target_outputs = {}
   ...: 
   ...:         def forward_hook(_, __, output):
   ...:             self.target_outputs[threading.get_ident()] = output.detach()
   ...:         self.model.features[2].register_forward_hook(forward_hook)
   ...: 
   ...:     def forward(self,input):
   ...:         self.model(input)
   ...:         thread_id = threading.get_ident()
   ...:         result = self.target_outputs[thread_id]
   ...:         del self.target_outputs[thread_id]
   ...:         return result
   ...: 
   ...: 
   ...: model = vgg19()
   ...: model = model.cuda(0)
   ...: wrapper = Wrapper(model)
   ...: 
   ...: devices = [0, 1]
   ...: 
   ...: inputs = torch.randn(60,3,224,224).fill_(0).cuda(1)
   ...: out = torch.nn.parallel.data_parallel(wrapper, inputs, devices)
   ...: print("first forward:   ", out.shape)
   ...: inputs = torch.randn(60,3,224,224).fill_(1).cuda(1)
   ...: out = torch.nn.parallel.data_parallel(wrapper, inputs, devices)
   ...: print("second forward: ", out.shape)

The result is
first forward:    torch.Size([60, 64, 224, 224])
second forward:  torch.Size([60, 64, 224, 224])