Because I was debbuging it I put batch size and other parameters as low as possible to avoid filling my Ram (batch at 2). If I get bigger parameters the problem is much more visible.
When launching the model, it take ~5Gb in Ram but then slightly but steadily increase with each batch.
For now it’s just 5 Residuals blocks (so 10 Convs) plus some attention (3 Convs) so the model should fit into my Ram (16Gb)
I’m guessing I messed up with something, but I can’t find what and I never had such problems with pytorch before