Hello! I have the next problem: my torch model slows down by the end of an epoch and starts to perform well in a new epoch.
So, I use tqdm to measure iter/second performance and have the next picture: at start of training performance is about 20 iter/sec, but it slows down with increasing iteration number and finished with about 4it/sec. In new epoch it start again from 20 and return to 4 by the end. I remove all from training loop, now that is like in pytorch tutorial, no any additional steps.
Use Dataloader with num_workes>1 + pin_memory, I have about 250 GB of data, which pass to Loader init, so my Dataset just returns index, no any loading data from disk and no transformations, all data stored in init in the way it passed to forward.
How can i find this bottleneck of my model?