If you only want to do a forward pass and have all ressources released, you can create the input Variable with volatile=True.
If you want to backward, you need to keep intermediary results on everything that you are going to backward.