The amount of memory required for backward depends linearly on the depth of the network.
Pytorch does some optimizations to reduce the memory requirements, but it still needs a bit less than 2x as much memory as if you are computing only the forward (with volatile=False).