For your questions:

Use all reduce method 23 to communicate across processes;
Yes. And if you want to distributedly conducting evaluation, just follow how the example deal with training data. e.g. create a test_sampler for distribute data into GPUs.