I’ve traced the source of the error to cudnn/rnn.py during the forward pass that starts in line 190. The error comes from this part of the code:
190    def forward(fn, input, hx, weight, output, hy):
191        with torch.cuda.device_of(input):

        (...)

264            if tuple(hx.size()) != hidden_size:
265                raise RuntimeError('Expected hidden size {}, got {}'.format(
266                    hidden_size, tuple(hx.size())))

where looks like hidden_size is the properly formatted data, and hx is the actual data being passed to each GPU (there’s a similar snippet in line 370 but I think this is the relevant one because its during forward pass).
The offending object is hx, which is passed to forward as an argument, so it looks like forward expects hx to be properly split already by some previous process?