I think your general approach is correct. Alternatively to the padding you could also try to create batches with the same image_seq_lens, but depending on your data distribution this might not be easy.


You should see a slightly lower performance for the packing and unpacking, but if there is no way around this, you wouldn’t have a baseline to compare against.


One thing I’m a bit worried about is the usage of the .data attribute.
You’ve mentioned that you are using packed_actions.data and packed_actions_gt.data to compute the loss, which would detach these tensors (especially the model output) and you should see an error.
Could you remove the .data usage and try to calculate the loss directly using these tensors?