Thanks for the reply.
I tried to isolate the network code from the training pipeline to create a code snippet.
It seems like without the training pipeline it would perform normally.
One thing I did not mention is that I was using PyTorch-Lightning as the base structure.
Could it be a possible cause?
I am running on a single GPU, so I am not sure what you meant by synchronization issue.
Or you meant the synchronization between CPU and GPU?
Edit:
I just found out that I was using CPU for the test. If I switch to GPU, then the error would appear again.