Well, turns out this is not a memory leak, I am just running out of memory because the model takes about 13Gb of memory while training (on cpu) and sometimes gets killed depending on what else is running on the machine.
I was considering getting a Nvidia Gtx 1080 for speeding things up, do anyone has some experience training Resnets on that GPU and will the 8Gb version of this board be enough?