I ran torch.utils.bottleneck to try and identify the cause, but I am not sure if this is informative enough:
Without pack_padded_sequence:

image1149×987 24.6 KB

With pack_padded_sequence:

image1114×781 23.8 KB