The problem that I’m having is the following, when I specify the neural network’s weights and biases as “requires_grad=true” then the evaluation of my model uses around 16GB of memory (all of the GPU’s memory) but when I use “requires_grad=false” the model only uses around 4-5GB of memory, my question is basically whether  “requires_grad=true” using 3X more memory is quite typical and is something to be expected or whether it is a sign that things are being done very inefficiently?
My model is somewhat “non-conventional” in that there is a small neural network whose outputs are feed into a physical model (the physical part of the model is what causes the memory to spike).  Basically the outputs of the NN are operated on by a series of non-linear operations using various large tensors (~1.5GB each), is it likely that the model is just complicated and therefore the 3X is reasonable or is this a surprising result.
I’ve already checked https://pytorch.org/docs/stable/notes/faq.html 1 and I feel that I’m doing what is suggested there but I figured that I would check here as well just in case. Also I can post code if needed but if 3X seems reasonable then I will just have to live with it.
Thanks!