lelouedec:

when my memory usage is around 90% my training goes quite fast, and when it drops it lslow down so much the process.


I suppose the GPU utilization is dropping from 90% to 10% or is it really the memory?
Could you elaborate a bit more about the batching code in front of the training?
I don’t really understand it, but maybe you could just just a Dataset and DataLoader to speed thing up?