Well, the problem is that the network I build above is the backbone of a detection model, so the concatenated feature maps will be used for downstream tasks (detection, segmentation, etc.), before which the loss cannot be calculated and thus I think I am not able to get the gredient and accumulate them. The desire behavior of the backbone is that when I am forwarding the cropped patches of one original input, the backbone will share the parameters (that is to say, no matter which patch the network is processing, the backbone uses the same parameters, just a bit kind of like a RNN). Under such behavior shall I reduce memory usage? And I wonder what kind of changes I should make to the code to achieve such behavior?