I’ve been thinking about your suggested solution again and it does actually solve my issue :). I just didn’t think the torch.cat() operation through properly and have been too stuck on the resize_() for some reason. It works with both functional and non-functional API.
so to have an example that works in my previous code’s style without the explicit copy operation for anyone reading this post:
weight1 = torch.randn(2,2,requires_grad=True) 
weight2 = torch.randn(2,2,requires_grad=True)
>>> weight1
tensor([[ 0.5209, -0.7963],
        [ 1.9585,  0.3662]])
>>> weight2
tensor([[-0.9158,  0.3723],
        [-1.1794,  0.1901]])

weight1.data = torch.cat((weight1.data, torch.randn(1,2)), dim=0)
>>> weight1
tensor([[ 0.5209, -0.7963],
        [ 1.9585,  0.3662],
        [ 0.6279,  2.3723]])
weight2.data = torch.cat((weight2.data, torch.randn(2,1)), dim=1)
>>> weight2
tensor([[-0.9158,  0.3723,  1.0036],
        [-1.1794,  0.1901, -0.6061]])

Thanks again @ptrblck =)