Yes, you’re right!
In the above test, I set cudnn.benchmark to True.
I just tried setting it to False. The memory cost drops.
Now, using VGG16, the maximum batch size increases to 60.
batch size    |  48     |   52   |   56    |    60
pytorch train | 7657 |  8115 |  7409 |  7659
Thank you! @smth