I use tensor.repeat() 32 to create replicates for each sample in the batch.
In particular, given hidden0 = torch.zeros(n_layers, 1, hidden_size), I use hidden0.repeat(1, B, 1) where B is the batch size.