Hi,
In all of resnets and even in your resnet50 (although you have removed some downsampling and pooling layers plus changed first layer to accept images with 1 channel), images get smaller and smaller and the reason is the definition of Conv2d. In each convolution, your input dimension let’s say x will be reduced to floor((x - kernel_size + 2* padding_size) / stride) + 1 and pooling layers just are a division by stride. So, What I meant is that if your model works for [x, x] input (batch_size does not matter as you said so), it must also work for [y, y] where y > x.
If you could share your model definition and stacktrace of error, it would be great as currently, it is hard to follow and apply some tests.
Also, about the definition of original resents, all of those accept inputs equal or bigger than [224, 224]. Less than this will lead to [batch_size, 0, ...] which literally is a zero size tensor. That is why I cannot figure it our why your model is ok with smaller size.
bests