I don’t what solution for this problem is available:

Write a sub-class of DataParallel for this ?
Should I reorganize my batches in another way such that using padding (would like to avoid this approach as it takes GRAM) ?
Forget the repeat_interleave and re-do the computation for each doc_embs (heavy computation).

The gain of multi-gpu would quitte consequent for this use-case.
Thank you for your help !