Solved by tom in post #2 
              
              
                What you describe is called NestedTensors in PyTorch parlance and it isn’t there yet. 
Until that materializes, the options are mostly padding (to regular shape) or packing. Either might be good to combine with “stratifying” your batches to have similar sizes, e.g. torchtext does this, I have done t…