To 1):
The code itself looks good. Why do you have to copy back to CPU and then to GPU again?
Also note that .resize_ doesnt initialize the values, so maybe you would like to fill the new filters with any values? I’m not familiar with your method, so maybe your method will be sufficient.
I think the code might be easier, if you use the functional API.
I created a very simple example just for one layer:
# Setup
weight1 = torch.randn(6, 1, 3, 3, requires_grad=True)
x = torch.randn(1, 1, 10, 10)
optimizer = optim.SGD([weight1], lr=1e-3)

# First training step
output = F.conv2d(x, weight1, bias=None, stride=1, padding=1)
output.mean().backward()
print(weight1.grad)
optimizer.step()
optimizer.zero_grad() # not necessary if you change weight1 immediately

# Add additional filter and train
weight1 = torch.cat((weight1.data, torch.zeros(1, 1, 3, 3)), dim=0)
weight1.requires_grad_(True)
optimizer = optim.SGD([weight1], lr=1e-3)
output = F.conv2d(x, weight1, bias=None, stride=1, padding=1)
output.mean().backward()
print(weight1.grad)
optimizer.step()
optimizer.zero_grad()

To 2):
I think you end up recreating the optimizer for each weight change, if you use this approach.
At least I’m not aware of an easy approach of updating the optimizer.