In addition to what have been discussed so far, I found that adding this line of code:
cudnn.benchmark = True

before the training takes place, will improve the speed performance if you are using GPU(s).