It it totally possible to use those new input shapes, because the forward method calls a adaptive_average_pool function before the Linear layer is called and therefor there should be no shape issues!
The performance will be a bit slower, because there are more dot products being calculated, but it should not be magnificent!