First of all thanks for your answer.

Yes the sequence length is about 3000 for all input samples
I cannot run Pytorch on the GPU, so the device available is a 2,6 GHz 6-Core Intel Core i7

As I never worked with packed sequences in PyTorch, I was not sure if the packing and unpacking of the sequences is the bottleneck of the training. However, as I now tested my pipeline on another dataset with much shorter sequence length (and it worked much faster), I assume that my implementation is alright.
Thanks for your help!