Have a look at this post 126, which explains this issue due to Python’s function scoping.
You might want to warp your training and evaluation in separate methods so that tensors can be freed when you return from these methods.