I have been experimenting with every possible number of workers. Using a GPU (TitanX, Ti1080) and CPU Training. The drops occur before/after every epoch. Currently an epoch takes around 8 s for images of size 480x640 and a batch size of 20.