I want to point at something that you might have missed: you update the  time_data  meter with the  lagrange  variable, before updating it again with  t5-t1 . How does your  meters  work? Does it return some average or the last value? This could influence the printed  time_data  at the end…

Good catch!, I just fixed that. It did even things out slightly more, but time_tot is still 30 times more than time_data (and that includes the factor 6 for the batch vs epoch).

Also, do you really need 20 workers for such a small dataset? Have you tried different values?

I started out with 2 workers, and just ramped it up when I noticed the delay, but it doesn’t really seem to make any difference.