Is there a way to have one weight matrix that is partly trainable and partly not?
Im trying to implement a RNN module that incorporates multiple time scales (as proposed in A Clockwerk RNN 1) but struggle on weight matrices that are partitioned in blocks of trainable weights and untrainable zeros (W_H in the image below).

Bildschirmfoto 2019-08-12 um 14.30.49.png980×380 24 KB

I tried to just initialize the Parameters as individual blocks and concatenate them, but using torch.cat on Parameters doesn’t work.
Is there a nice way of handling this or do I have to do all calculations separately?