Another approach would be to use an adaptive pooling layer 6, which outputs an activation volume using the spezified output_size. This would allow to feed differently sized input (same size in the batch) through the model and create a fixed sized activation before the linear layer.
However, you would need to group same sized image into batches.