Thank you for explanation.
I get all the numbers from nvidia-smi. I have tried out using torch.cuda.empty_cache() to free all unused memory and I get the same result as I stated before.
I know that memory allocation will be different on different architectures, but is it responsible for such a huge difference of memory usage, i.e. 5286MB v.s. 7539MB, on two kinds of GPUs?
Also, I do not understand why the algorithm consults much more memory since the second backward compared with the first backward on TITAN X, while it consults the same memory no matter which time backward on GeForce GTX.