One of the things this flag controls is whether to broadcast model buffers in each iteration to ensure that module buffers are synchronized across processes. You can evaluate the speed up of your model with broadcast_buffers=False and remove it if model accuracy permits.