@rvarm1, Thanks for you response!
In my case, batch size=1 which follows to one frame per GPU, gradient accumulation interval include all patches (one frame is set of multiple patches). Basically, multi-gpu allows to speed up training time, which is main bottleneck for my task.
The training script is configured to support multi-gpu and single-gpu with torch.multiprocessing.spawn. I tried to train with 1 gpu, training works well. But for multi-gpu, it fails and train/val loss starts increase after few epochs. The difference between successful and failed cases is only sync.
I am also tried different learning rate, there are no any changes.