I have tried to achieve the similar feature like yours, and I encountered similar problem.
I guess the major problems are as follows:

In the inference phase or BP phase, some tensors/grads can be deleted when corresponding specific computations are finished. However, in the beginning, pytorch does not know when to delete which tensor/grad, as it is a dynamic-graph framework, so the memory cost is high. After several passes, pytorch knows the architecture of CNNs, and delete tensors/grads as soon as possible in subsequent passes, so the memory cost is low.
PyTorch chooses base computation method according to batchsize and other situations, so the memory cost is not only related to batchsize.