Could you check, if your system is maybe overheating and might reduce the clocks of your hardware?
If that’s not the case, could you profile which part of the code is causing the slowdown?
You could use the timers from the ImageNet example to profile the data loading etc.