wqn628:

input_len


It seems you are doing the sort/unsort stuff for sequence packing out of the forward method, which you should do inside based on your input (param #1 for forward method). What happened here is, each of your GPU got 4 sequences, with a sorted length of 12 sequences.