Do you happen to know what are the breaks in the GPU processing? GPU doesn’t do anything for a little and then starts working again (works in cycles).

I guess updating the weight is probably not requiring much computing. Or other tasks that you may have in your network where you are copying values etc. Essentially everything that is not a large dot product or matrix multiplication.