Can you try this?
def __init__(self):
    self.hidden_params = torch.nn.Parameter(torch.randn(self.layers * directions,  1, self.hidden_size))
    self.cell_params = torch.nn.Parameter(torch.randn(self.layers * directions, 1, self.hidden_size))
    ....

def forward(self, x):
    hidden_params = self.hidden_params.repeat(1, batch_size, 1)
    cell_params = self.cell_params.repeat(1, batch_size, 1)
    ... = self.rnn(inp, (self.hidden_params, self.cell_params))
   

Basically, try to repeat the dimensions in forward call.