You have to pay attention to the weight_hh  and weight_ih, because they are a concatenation of four different sub matrices. Thus the desired behaviour you want isn’t actually happening. You will have to do a init.orthogonal_(sub_matrix) for each four of them.