The main point is that the eight matrices of lstm are grouped into two groups, reason being performance, so you have “weight_ih”, which is a concatenation of “(W_ii|W_if|W_ig|W_io)” and “weight_hh”, is of “(W_hi|W_hf|W_hg|W_ho)”. So you do the orthogonal initialization to the sub matrices of “weight_hh” and the xavier to the sub matrices of “weight_ih”.