My understanding is due to pytorch automatically select the cudnn algorithm, when there is adequate memory, it just use more. So it may not be fair to compare memory cost without considering the cudnn alogrithm.
In my comparison above, you can see batchsize=52 cost more memory than batchsize=56, because my memory limit is 8G. So I think one reasonable comparison metric could be ‘the maximum batch size given xx memory size’.