There are 4 ways to deal with variable size inputs:

one by one training (batch size = 1).
resizing (or up/down sampling) the samples to the same size in a batch.
crop the samples to the same size in a batch.
padding the samples to the max size in a batch, and masking the redundant parts after every y=f(x).


I would really like to avoid padding my data (in both “dimensions”), as just the back of the envelope calculation suggests the padded size will be at least 2x the raw data set size. I just think the overhead of transferring all of that data to the GPU is going to be a massive waste of time.

Why there is a “2x”?