Now, it make sense. There was a small issue which was causing to increase the memory. I have updated my code (train_mtl.py and util.py). Request you to take latest and run it again from scratch.
Thanks