I am experiencing the same issue. When I switch to validation and switch back, the beginning few hundred iterations is always very slow, about 4~5 times slower than normal. Some example statistics from the training process:
2018-10-17 20:12:12 Validation: max_iter: 1725, loss:1.191111, Validation cer: 0.065339, accuracyCTC: 0.625263
2018-10-17 20:16:01 Train: [26/150][430500/2598450] Loss: 0.408760 Loss_CTC: 0.408760
time elapsed 1747
2018-10-17 20:19:37 Train: [26/150][431000/2598450] Loss: 0.353606 Loss_CTC: 0.353606
time elapsed 216
2018-10-17 20:23:46 Train: [26/150][431500/2598450] Loss: 0.396339 Loss_CTC: 0.396339
time elapsed 249
2018-10-17 20:30:23 Train: [26/150][432000/2598450] Loss: 0.379618 Loss_CTC: 0.379618
time elapsed 396
2018-10-17 20:35:08 Train: [27/150][432500/2598450] Loss: 0.395716 Loss_CTC: 0.395716
time elapsed 285
2018-10-17 20:40:20 Train: [27/150][433000/2598450] Loss: 0.430345 Loss_CTC: 0.430345
time elapsed 311
2018-10-17 20:45:49 Train: [27/150][433500/2598450] Loss: 0.451907 Loss_CTC: 0.451907
time elapsed 328
2018-10-17 20:49:50 Train: [27/150][434000/2598450] Loss: 0.393155 Loss_CTC: 0.393155
time elapsed 240
2018-10-17 20:54:10 Train: [27/150][434500/2598450] Loss: 0.375856 Loss_CTC: 0.375856
time elapsed 260
2018-10-17 20:59:41 Train: [27/150][435000/2598450] Loss: 0.374424 Loss_CTC: 0.374424
time elapsed 330
2018-10-17 21:24:42 Validation: max_iter: 1725, loss:1.233052, Validation cer: 0.065245, accuracyCTC: 0.623922
2018-10-17 21:29:18 Train: [27/150][435500/2598450] Loss: 0.400490 Loss_CTC: 0.400490
time elapsed 1777
2018-10-17 21:34:05 Train: [27/150][436000/2598450] Loss: 0.559940 Loss_CTC: 0.559940
time elapsed 286
2018-10-17 21:38:16 Train: [27/150][436500/2598450] Loss: 0.469486 Loss_CTC: 0.469486
time elapsed 250
2018-10-17 21:42:23 Train: [27/150][437000/2598450] Loss: 0.460522 Loss_CTC: 0.460522
time elapsed 246
2018-10-17 21:46:36 Train: [27/150][437500/2598450] Loss: 0.417255 Loss_CTC: 0.417255
time elapsed 252
2018-10-17 21:50:29 Train: [27/150][438000/2598450] Loss: 0.446802 Loss_CTC: 0.446802
time elapsed 233
2018-10-17 21:54:54 Train: [27/150][438500/2598450] Loss: 0.405387 Loss_CTC: 0.405387
time elapsed 264
2018-10-17 22:00:10 Train: [27/150][439000/2598450] Loss: 0.388070 Loss_CTC: 0.388070
time elapsed 316
2018-10-17 22:04:14 Train: [27/150][439500/2598450] Loss: 0.399813 Loss_CTC: 0.399813
time elapsed 243
2018-10-17 22:08:08 Train: [27/150][440000/2598450] Loss: 0.386462 Loss_CTC: 0.386462
time elapsed 233