Solved by JuanFMontesinos in post #2 
              
              
                Variables not only store the data you typically use but also a context graph which indicates how to backpropagate gradients to leaf variables. 
If you detach your model you are getting rid of that context, thus, using less memory.