I just made a test now.
Epoch 1:
train iter 1: 6139M
train iter last: 7209M
eval iter 1: 3291M
eval iter last: 3291M
inference iter 1: 3291M
inference iter last: 3291M
Epoch 2:
train iter 1: 6409M
train iter last: 7477M
eval iter 1: 7479M
eval iter last: 7479M
inference iter 1: 7479M
inference iter last: 7479M
Epoch 3:
train iter 1: 7479M
train iter last: 7479M
eval iter 1: 7479M
eval iter last: 7479M
inference iter 1: 7479M
inference iter last: 7479M
Maybe this is normal in which case i learned something new. My guess is that there is a “knowledge” of the pattern train>eval>inference and the gpu stores in memory the computational graph even if we are in a torch.no_grad() env for the next epochs?