We never store all intermediate variables, but only the ones that are absolutely necessary to compute the backward. Everything else is freed immediately.
What kind of modules/operations are you using in your network? The increased memory usage might be coming from the fact that we’re always trying to pick fast algorithms in cuDNN, even at a cost of memory usage. We’ll be trying to make it possible to affect these choices in the future.