Did you try more than one iteration? Because when I use 2 GPUs with 3 iterations I get the following:
 gpu 1  calculate_running: True
 gpu 0  calculate_running: True
 gpu 1  calculate_running: True
 gpu 0  calculate_running: True
 gpu 0  calculate_running: True
 gpu 1  calculate_running: True

But I want to see this:
 gpu 1  calculate_running: True
 gpu 0  calculate_running: True