Please let me know if the version is a problem

Yes, the memory leak could be fixed by new PRs landed after v1.4 (v1.4 branch cut was 12/08/2019 IIRC). Can you try the master branch, or the nightly (conda install pytorch torchvision -c pytorch-nightly), or if you can share a self-runnable script, we can try that too.

if you could share more thoughts on the code I provided above.

The code you share above looks correct to me.