Thanks for your feedback.
To 2) I guess this will just have to do then. If there isn’t any apparent pitfalls where this can go wrong then the overhead shouldn’t be too bad.
To 1). I think I should have given a direct example to make it clear. Your solution with functional API and torch.cat() looks nice but I don’t think it addresses the issue here as your example only deals with one layer (or I’m maybe totally just missing it? ;)). The challenge really comes in when dealing with layer/weight2. This is also why I need a copy. My bad for not giving a concise easy to reproduce example. So here it is:
For simplicity let’s assume these are the weights of two consecutive linear layers where we start with fan_in and fan_out = 1 in each layer (because the spatial dimensions don’t really contribute to the argument). I’m going to leave out the SGD parts because that seams to be solved.
weight1 = torch.randn(2,2, requires_grad=True) 
>>> weight1
tensor([[-1.1513, -1.6009],
        [ 0.5701,  0.4365]])
weight2 = torch.randn(2,2, requires_grad=True) 
>>> weight2
tensor([[ 0.7709,  1.9652],
        [ 1.7878, -0.3036]])

According to some criteria we now add a unit/filter to weight1. This is easy and I believe your cat solution could also be used here.
weight1.data.resize_(3,2)
tensor([[-1.1513e+00, -1.6009e+00],
        [ 5.7011e-01,  4.3649e-01],
        [ 8.8747e-29,  4.5771e-41]])
weight1[-1:,:].normal_()
>>> weight1
tensor([[-1.1513, -1.6009],
        [ 0.5701,  0.4365],
        [-0.5734,  0.3861]])

So as we can see we can use resize_() in dimension 0 without modifying previous values (it needs to be on .data as require_grad params can’t be resized). I had left out initialization for simplicity before. But of course you are right that we need to init the corresponding new slice as otherwise there will be junk values in the weights.
But thats only because we are resizing dimension 0! Now if you think of this as two following layers, this change in weight1 implies a change in weight2 in the other non-spatial dimension , because fan_out of one layer is fan_in of the other. And yes I know it might look confusing, but it looks like in PyTorch fan_out is actually dim=0 and fan_in is dim=1. If you think of it from a flattened point of view it makes sense though as fan_in would get flattened together with spatial dimensions dim=2,3 in computation of convolutions.
If we naively go and do the same code, giving that we are modifying dimension 1 instead of 0, we are going to mess up previous information:
>>> weight2
tensor([[ 0.7709,  1.9652],
        [ 1.7878, -0.3036]])
weight2.data.resize_(2,3)
>>> tensor([[ 7.7089e-01,  1.9652e+00,  1.7878e+00],
        [-3.0362e-01,  8.2830e-29,  4.5771e-41]])

It’s already quite clear here that the information is no longer the same (because of internally flattened representations probably). The weights are not preserved here.
So the only work-around I see for this is to do something like this:
tmp = weight2.data.clone()
weight2.data.resize_(2,3)
>>> tensor([[ 7.7089e-01,  1.9652e+00,  1.7878e+00],
        [-3.0362e-01,  1.6501e+19,  4.0092e-08]])
weight2.data[:,0:-1] = tmp
>>> weight2
tensor([[ 7.7089e-01,  1.9652e+00,  1.7878e+00],
        [ 1.7878e+00, -3.0362e-01,  4.0092e-08]])
weight2.data[:,-1:].normal_()
>>> weight2
tensor([[ 0.7709,  1.9652,  0.8498],
        [ 1.7878, -0.3036,  0.2938]])

Which is what we actually want. So here you can see why I need a temporary copy and why I transfer to CPU. Because if you imagine actually talking about a VGG/ResNet/Segnet or some other memory intense network, I wouldn’t really want to clone the entire weights just to resize the weights.
I hope this makes it more clear. I will appreciate any further feedback!