Thank you very much. How about the following:
w = nn.Parameter(torch.Tensor(4,4))
rnn1 = nn.RNN(3,4)
rnn2 = nn.RNN(3,4)
rnn1.weight_hh_l0 = w
rnn2.weight_hh_l0 = w

I guess it could also work. And another one that I am not sure wether the autograd for w involves rnn1 and rnn2. Does this  dw = drnn1(w) + drnn2(w)  hold true when I am taking the gradient of w with pytorch autograd?