I just tested it with 0.3.1.post2 (on CPU only) and my memory was growing up to 38GB.
Could you try to build PyTorch from source? Maybe it’s a leak, which was recently fixed.
You can find the build instructions here 7.
It should be quite easy to compile it, but let me know, if you encounter any issues.