Hi again, I did some experiments with batch_size 4, 6 and 8 and the OOM always occurs at the same number of iterations for each batch size: for 4 at 1288 iterations, for 6 at 372 ,for 8 at 102. Something is very strange, I don’t know where form comes this fix limitations. The memory type that blows out is the cached GPU memory, it reaches a 3 or 4 times the normal threshold:
93it [00:36,  3.31it/s]stack - cached memory before: 610.75
94it [00:36,  3.13it/s]stack - cached memory before: 618.0
95it [00:36,  2.99it/s]stack - cached memory before: 625.5
96it [00:37,  2.88it/s]stack - cached memory before: 616.25
97it [00:37,  2.72it/s]stack - cached memory before: 588.375
98it [00:38,  2.74it/s]stack - cached memory before: 571.625
99it [00:38,  2.81it/s]stack - cached memory before: 576.0
100it [00:38,  2.87it/s]stack - cached memory before: 583.375
101it [00:39,  2.89it/s]stack - cached memory before: 421.0
102it [00:39,  3.24it/s]stack - cached memory before: 2360.75