In test/eval phase you can create the input to your model as a Variable with a volatile flag - Variable(data, volatile=True). This will ensure that no buffers needed for backward or outputs (if you don’t save them anywhere) will be kept, and will be freed as soon as they can.
Training is already quite memory efficient. If it doesn’t need to hold onto some input/output because it can compute the gradient in another way, it won’t do it.