When I tried recreated in numpy I just copied for instance like this:
weight_ih_l0 tensor([[-0.1641, -0.6958],
       [ 0.1889,  0.4084]])

then
ih = numpy.matrix([[-0.1641, -0.6958],
       [ 0.1889,  0.4084]])

bias_ih = ([-0.0039, -0.2850])
and so on

But when I reshaped my bias and my input like this:
bias_ih = numpy.matrix([[-0.0039],  [-0.2850])
(and another bias)
h_t = numpy.tanh(hh@hidden + ih@numpy.matrix([[1],[1]]) + bias_ih + bias_hh) 

it is started to work correctly and match pytorch outputs, so is there auto-reshaper of some kind or pytorch have some kind of different printed shape(sorry for reinventing words :/)?