I run your code in my server------ it works fine. I run it for 25 epochs, the memory only rise at the start  of the 2nd epoch which rise from 527MB to 551MB, and then it keeps the same.
using GPU  torch ‘0.2.0_3’