Almost every model nowadays uses Adaptive pooling at the end of their model. You can also try training your model with different input size images, which would provide regularization.
You had 320x320 images. Now start your training at 80x80 resized images. Then use 160x160 resized images and train and then use 320x320 images and train. In this way you are providing 4x more pixels for your model to train on at each step. This technique requires you to use Adaptive pooling. Use a concatenation of AdaptiveAvgPool and AdaptiveMaxPool for the best results.