Thanks for your responses.
I will put the code maybe it is useful for others:
# Unwrap from DataParallel
    if torch.cuda.device_count() > 1:
        model = model.module

    model.classifier = Identity()

    if torch.cuda.device_count() > 1:
        print("Wrap Again with DataParallel with ", torch.cuda.device_count(),
              "GPUs!")
        model = nn.DataParallel(model)