Solved by tenrand in post #2 
              
              
                Ah,  just understood what is happening. I am storing loss in a list which is keeping the whole backward pass graph alive. I need to detach it before storing. 
Thank you everyone i will close this silly question.