Perhaps I am not making myself very clear, I will try my best to make it clear. The whole point is that, by cropping input image into patches, is there any possibility to reduce memory usage during the training?
As you have explained above, there is NO memory usage reduction because I have to concatenate the feature maps for downstream tasks, thus whether passing a whole image or passing patches will produce similar intermediate variants (intermediate activation tensors, etc.) for backward propagation, thus no memory usage reduction.
But, (if I understand it correctly), if every time I perform the backward process (calculate the loss and get the gradient), the intermediate activation tensors will be freed, thus reduing memory usage. But this approach might not work in my case as there is no suitable loss function to get the loss of my backbone’s output feature maps.
Thus, another idea of mine is that, what if I modify the backbone into a RNN? As the backbone actually takes the patches one by one, a serial input, I think it is really like a RNN. So, I am curious that, if I modify my network to make it become a RNN, could such modification reduce memory usage?