I’m not sure I understand the issue completely, since you were mentioning a memory increase in the first part, but also explain that the memory usage reaches a certain level for a smaller batch size.
Is this memory peak stable for the smaller batch sizes or does it also increase for multiple training and validation epochs?