Oh thanks I will try that torch.no_grad()
During source training, the size is 2000 for min_size and 4500 for large_size, with which it would not run out of memory. During transfer training, min_size is 1200 at most.
My images are drawn from remote sensing images which are quite large (about 2k * 2k) but the objects are reletativly small (about 100 * 100) and there are some tiny textures on the object like the lines distinguishing tennis court from basketball court, so reducing the size would reduce the performance.