I use adaptive_avg_pool2d to deal with the size differences. And instead of a classical linear layer I use a convolution layer with a kernel size of 1, which acts like a dense layer (as proposed in the paper Network in Network).
The full stucture is like this:
Conv -> BatchNorm -> Relu -> Conv (Stride > 1) -> Relu -> … Conv (64, 64, Kernel = 1) -> BatchNorm -> Relu -> Conv (Stride > 1) -> Relu -> Conv (64, NUM_CLASSES) -> BatchNorm -> Relu -> adaptive_avg_pool2d -> Softmax
This structure is derived from the paper “Striving for simplicity: The all convolutional net” and learns quite well.
But the suggestion to group same sized images is a good idea I might pursue. Still I would like to understand why feeding the samples one by one collecting the outputs does not work. Could you maybe explain

Since you are feeding the samples one by one, the running estimates might be quite off.

a little further?