hsiangyu:

Thus, another idea of mine is that, what if I modify the backbone into a RNN? As the backbone actually takes the patches one by one, a serial input, I think it is really like a RNN. So, I am curious that, if I modify my network to make it become a RNN, could such modification reduce memory usage?


I don’t think so, as it would create the same computation graph in the end but in a “sequential” way instead of multiple forward “parallel” graphs. Autograd would thus backprop through all “steps” and use the same amount of memory.
Again, I would recommend to check the needed activations for your ideas and compare it to the baseline of using the complete image.