I tried using the same approach that you mentioned for dynamically changing weight matrix.
But I am getting error as
RuntimeError: size mismatch, m1: [771 x 20], m2: [3 x 20] at /opt/conda/conda-bld/pytorch-cpu_1532576596369/work/aten/src/TH/generic/THTensorMath.cpp:2070
In the code I have to actually add atleast 12 hidden neurons dynamically but initially I tried using only one neuron. (trying cascade architecture with RPROP)
Code:-
class MultiLayerNet(torch.nn.Module):
def init(self, n_input, n_hidden, n_output):
super(MultiLayerNet, self).init()
self.sigmoid = torch.nn.Sigmoid()
self.relu = torch.nn.ReLU()
self.weight1 = nn.Parameter(torch.randn(n_input, 3)) # for all hidden layers
self.weight2 = nn.Parameter(torch.randn(2, 2)) # for output layer
def forward(self, x,count):
    print(count)
    print(x.shape, self.weight1)   
    if count == 1:
        h_pred = self.relu(F.linear(x,self.weight1))
        
    y_pred = self.sigmoid(F.linear(h_pred,self.weight2))
        
    return y_pred

Note:-
input as x is of shape 771 x 20 and weight1 matrix is of size 20 x 3 which should actually get easily multiplied but I observed that in linear function instead of directly using the passed weight1 matrix  it is using weight.t1() (transpose of the matrix) not sure why?
Further logs:-
 in forward(self, x, count)
19         print(x.shape, self.weight1)
20         if count == 1:
—> 21             h_pred = self.relu(F.linear(x,self.weight1))
22
23         y_pred = self.sigmoid(F.linear(h_pred,self.weight2))
/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
1024         return torch.addmm(bias, input, weight.t())
1025
-> 1026     output = input.matmul(weight.t())
1027     if bias is not None:
1028         output += bias