Hi all,
I have a problem about memory consulting on different GPUs. I implement a model containing convolution layers and LSTM. I try to train it using both the GPU on my workstation and also the GPU on the server. However, it consults different size of memory on different GPUs, which confuses me.
The GPU on my workstation is GeForce GTX. When the model uses this GPU, it takes 5286MB in total. However, when it runs on the TITAN X on the server, it takes up to 7539MB. There is a huge difference.
Especially, when it runs on TITAN X, it consults only 5315MB after first time backward and it consults 7539MB since the second time backward. I do not understand why the second time backward consults so much memory? Since the memory usage holds the same after the second backward, I cannot conclude there exists memory leaks.
Can anyone share any thought about this situation?
Best