Could you check if you are appending some tensors to a list and could thus be storing the computation graphs throughout the epoch? This should be visible in an increased memory usage in the first epoch. Also, are you using backward(retain_graph=True) or any other “special” setup?