Thanks for the repro! There is a bug somewhere here, would you mind filing an issue on GitHub 11?
For some reason it works for me on master if you use PackedSequence directly instead of the qualified version, so that could be a workaround until we get this fixed:
    def _pad(self, data, batch_first: bool, batch_sizes, pad_value: float, sorted_indices: Optional[torch.Tensor], unsorted_indices: Optional[torch.Tensor]):
        packed_seq = PackedSequence(data, batch_sizes, sorted_indices, unsorted_indices)
        return torch.nn.utils.rnn.pad_packed_sequence(packed_seq, batch_first, pad_value)