It depends if you are training or testing.
In training, the internal layer outputs and other gradient biffers occupy space.
During testing, you could use torch.no_grad() to reduce memory consumption. Possibly try with this.