You are probably doing something wrong then. In usual cases, memory usage shouldn’t increase as you run more iterations. This is good starting point to debug: http://pytorch.org/docs/master/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory 287