Thanks for your response. If I do like you suggest all my gradients die:
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]]), tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]), tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]), tensor([0., 0.])] ```

Original order:

```[tensor([[-0.9561,  0.0000,  0.0000,  ...,  0.0000,  0.0000, -0.1120],
        [-0.0981,  0.0000,  0.0000,  ...,  0.2065,  0.0000, -0.0217],
        [-0.4990,  0.0000,  0.0000,  ..., -0.2006,  0.1060,  0.0247],
        ...,
        [ 0.1178,  0.0000,  0.0000,  ...,  0.0000,  0.0000, -0.0803],
        [-0.0503,  0.0000,  0.0000,  ...,  0.0000,  0.1357,  0.0162],
        [ 1.3231,  0.0000,  0.0000,  ...,  0.5383,  0.0000,  0.3681]]), tensor([-1.5053, -0.6233, -0.2615, -1.0045,  0.3973,  0.6638,  1.0489, -0.1003,
        -0.2944,  1.3231]), tensor([[ 0.8244, -0.5169, -1.6222, -1.2467,  2.6140,  1.1356, -0.7496, -0.6147,
          0.7515,  1.6823],
        [-0.8244,  0.5169,  1.6222,  1.2467, -2.6140, -1.1356,  0.7496,  0.6147,
         -0.7515, -1.6823]]), tensor([-3.6501,  3.6501])] ```

Regardless of depth and hidden size.