You can partially choose cpu or gpu for each weight. Did you mean this?
model = models.resnet18()
model.cuda()
model.layer1[0].conv1.weight.type()
'torch.cuda.FloatTensor'
model.layer1[0].conv2.weight.type()
'torch.cuda.FloatTensor'
model.layer1[0].conv1.cpu()
Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
model.layer1[0].conv1.weight.type()
'torch.FloatTensor'
model.layer1[0].conv2.weight.type()
'torch.cuda.FloatTensor'