So I played around with the timers in that code (which is awesome by the way, thank you) and also did a few little experiments with torch.save and torch.load. I tried to be slick and save the part which took the longest each time so that I could just pull it back up and train. Yeah, it doesn’t work that way.
What I did learn though was that my code takes the longest time when I do the following:
loss_dict = model(imgs, annotations)
When that part get’s executed it takes almost all my memory and things start to slow down. Maybe I should mention that I have a Nvidia Xavier. It have the 32GB model with a 512GB drive attached (this is where I store all my stuff). My dataset is about 8600 images (512 x 640). It’s a COCO dataset and I’m using a Faster RCNN - Resnet50.
It takes me about 30 hours to train this model. Perhaps that’s just what to expect from my gear but looking at others examples, somehow it just seems like this is slow. If it is slow then I’ve definitely done something wrong in my code.
Am I just expecting more than I should?