Hi
I train a speaker recognition model using ResNet34 model (with little modifications). The code is available at github 2. At the beginning epochs, each epoch takes about 15 seconds:
image1920×1080 129 KB
After a while, time to complete an epoch becomes about 30 seconds:
image1920×1080 232 KB
I really can’t understand what is going wrong! There are huge amount of free ram and the GPU memory footprint does not change during this transition.
P.S. I saw this behavior after adding learning rate scheduler. I’m not sure this may relates to scheduler or not.
Another weird behavior is that killing and running the process (even after rebooting the system!) does not reset the epoch time to 15 seconds! The only solution I found effective is to power off and on my system!
Any help is appreciated.
regards