it looks like a gpu issue.
i run the same code over 4 different clusters that have different types of gpus.

cluster 1: p100.  3gb < gpu mem used < 6gb. memory usage completely stable. no issue.
cluster 2: v100.  3gb < gpu mem used < 6gb. memory usage completely stable. no issue.
cluster 3: p100.  3gb < gpu mem used < 6gb. memory usage completely stable. no issue.
cluster 4: t4.  6gb < gpu mem used < 16gb. memory usage completely insanely unstable. reached oom.

for 4th case, the memory usage starts at 6gb. than, after ~200 epochs of being stable, memory usage starts to oscillate between 6 to 8gb.
later, between 7 and 15gb… crossing 11, 12, 13, 14… gb
i use torch.cuda.empty_cache before and after validation.
i checked, there is no memory leak.
if there is one, it would appear in other cases as well. [there was one couple weeks ago, but i fixed it.]
any idea why t4  are behaving like this?
in all cases, i am using mixed precision training.
because they have plenty of tensor cores, i expect the memory usage to be the lowest.
i am also using ddp, but with single gpu with gloo backend.
the oom happens in the training phase.
pytorch 1.10.0 installation:
pip install torch==1.10.0 -f https://download.pytorch.org/whl/cu111/torch-1.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl
thanks