Hi,
I have a variant of inception v3 network, which is pre-trained and fixed. Now I want to do the inference for that network.
I tried both tensorflow and pytorch.
Tensorflow:  I saved the network as graphdef, and use it for inference. I can use 128 batch size for inference.
Pytorch: I put the model in evaluation mode, but I cannot use 128 batch size, which will give me out of memory error.
I don’t know why this happens. My guess is that tensorflow may not cache the intermediate feature maps in the graphdef mode, but pytorch may do.
I also had other problems related to the memory usage. My recent experience suggests that pytorch often uses higher gpu memory than tensorflow. In many cases, I have to reduce batch size, which may or may not solve my problem. Are there any suggestions on how to use pytorch more memory efficiently?