Solved by albanD in post #2 
              
              
                Hi, 
What you would want here is a NestedTensor. This is work in progress at the moment. You can follow the current status on this issue: https://github.com/pytorch/pytorch/issues/25032