qap:

This means that I could have 4 of these, which means training over 4 GPUs. Would this make training faster?


This is possible. Hope Figure 9 in this paper can offer some insight: https://arxiv.org/pdf/2006.15704.pdf 9

What factors should be taken into consideration?

If the GPUs are the same, the network bandwidth is one of the dominating factor of training speed.

4 instances do take more time to set up than 2.

This is one-time setup overhead, instead of per-iteration overhead, right? If so, it should be fine.