@ptrblck Thanks. I tried your guess, but it may not have much to do with parameter initialization, it seems that the initialization process is consistent.
and I just modify the size of input, h0, c0 tensor, the output is the same.
Ver 1.6
Python 3.8.5 (default, May 27 2021, 13:30:53) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.manual_seed(0)
<torch._C.Generator object at 0xffff8d5e78b0>
>>> lstm = torch.nn.LSTM(2, 3, 2)
>>> input = torch.randn(2, 2, 2)
>>> h0 = torch.randn(2, 2, 3)
>>> c0 = torch.randn(2, 2, 3)
>>> output, (hn, cn) = lstm(input, (h0, c0))
>>> h0
tensor([[[-0.6731, -1.2095,  1.3441],
         [ 2.3832, -0.5665, -1.1536]],

        [[-2.5023,  0.8756, -2.6726],
         [-0.0313,  0.4988, -0.5233]]])
>>> c0
tensor([[[-0.2515, -1.0555, -0.5593],
         [-0.1197, -0.1635, -0.2505]],

        [[-1.0882,  0.0044,  0.2328],
         [-0.8201, -0.6790,  1.4251]]])
>>> hn
tensor([[[ 0.3527, -0.1816, -0.0543],
         [ 0.0832, -0.2947, -0.0653]],

        [[-0.3593, -0.1131,  0.0362],
         [-0.2696, -0.1553,  0.0931]]], grad_fn=<StackBackward>)
>>> cn
tensor([[[ 0.4743, -0.4206, -0.2134],
         [ 0.1192, -0.5162, -0.1760]],

        [[-0.9582, -0.1660,  0.0946],
         [-0.6909, -0.2449,  0.2589]]], grad_fn=<StackBackward>)
>>> output
tensor([[[-5.7776e-01, -8.8637e-02,  5.1628e-04],
         [-3.1305e-01, -1.5366e-01,  2.0715e-01]],

        [[-3.5934e-01, -1.1314e-01,  3.6213e-02],
         [-2.6963e-01, -1.5533e-01,  9.3064e-02]]], grad_fn=<StackBackward>)

Ver1.8
Python 3.8.5 (default, May 27 2021, 13:30:53) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1'
>>> torch.manual_seed(0)
<torch._C.Generator object at 0xffffaaf233b0>
>>> lstm = torch.nn.LSTM(2, 3, 2)
>>> input = torch.randn(2, 2, 2)
>>> h0 = torch.randn(2, 2, 3)
>>> c0 = torch.randn(2, 2, 3)
>>> output, (hn, cn) = lstm(input, (h0, c0))
>>> h0
tensor([[[-0.6731, -1.2095,  1.3441],
         [ 2.3832, -0.5665, -1.1536]],

        [[-2.5023,  0.8756, -2.6726],
         [-0.0313,  0.4988, -0.5233]]])
>>> c0
tensor([[[-0.2515, -1.0555, -0.5593],
         [-0.1197, -0.1635, -0.2505]],

        [[-1.0882,  0.0044,  0.2328],
         [-0.8201, -0.6790,  1.4251]]])
>>> hn
tensor([[[ 0.3527, -0.1816, -0.0543],
         [ 0.0832, -0.2947, -0.0653]],

        [[-0.3593, -0.1131,  0.0362],
         [-0.2696, -0.1553,  0.0931]]], grad_fn=<StackBackward>)
>>> cn
tensor([[[ 0.4743, -0.4206, -0.2134],
         [ 0.1192, -0.5162, -0.1760]],

        [[-0.9582, -0.1660,  0.0946],
         [-0.6909, -0.2449,  0.2589]]], grad_fn=<StackBackward>)
>>> output
tensor([[[-5.7776e-01, -8.8637e-02,  5.1628e-04],
         [-3.1305e-01, -1.5366e-01,  2.0715e-01]],

        [[-3.5934e-01, -1.1314e-01,  3.6213e-02],
         [-2.6963e-01, -1.5533e-01,  9.3064e-02]]], grad_fn=<StackBackward>)

Hope for help!