Based on that graph you’ve got 512GB of RAM and it’s ramping to 200GB over the epoch and then back to 0? I’m not necessarily seeing an issue here if your system is of that scale. I assume then you may also have 8-16+ cores and may be using a high worker count, in which case you’ve could have a number of processes queuing up batches for preprocessing and ramping the memory as they do so…
To figure out whether this is expected behaviour or an issue, number of workers is required, the size of images in addition to the batch size. Also with regards to memory usage analysis, more detailed views of virtual, shared, and physical per process. Maybe an output of htop, turn off threads with ‘H’, sort by memory usage…