I just have exactly same situation, thanks for the solution. Since I want to make the portion of the network that are made multigpu as much as possible, finally I changed the torch/nn/parallel/scatter_gather.py 's gather() function to be :
def gather(outputs, target_device, dim=0):
    """
    Gathers variables from different GPUs on a specified device
      (-1 means the CPU).
    """
    def gather_map(outputs):
        out = outputs[0]
        if isinstance(out, Variable):
            return Gather(target_device, dim=dim)(*outputs)
        if isinstance(out, OrderedDict):
            return OrderedDict([
                (k, Gather(target_device, dim=dim)(*(each[k] for each in outputs))) \
                        for k in out.iterkeys()])
        if out is None:
            return None
        return type(out)(map(gather_map, zip(*outputs)))
    return gather_map(outputs)

and in the network, during forwarding, it returns an OrderedDict, .e.g.
def forward(self, x):
    x = self.features(x)
    x_emb = self.emb(x)  # normal embedding head
    # below are small classification heads for all types of attributes
    x_attr_dict = OrderedDict([(attr_name, mod(x)) for attr_name, mod in self.attr_classifiers.named_children()])
    return x, x_attr_dict

in this way, the whole network can be make multi gpu version, the x_attr_dict 's result will also be gathered properly though it is an OrderedDict with keys are attribute names and values are features for each attribute.