I’m also currently investigating a multi-GPU contrastive setup and I’m currently debugging the difference between a 1 GPU full bs and x GPU bs/x setup to find the difference between the loss implementations.
Therefore, I would be very curious on how others approached this and with what you ended up in the end.