Several configuration I could think of:

Train and validate on all possible same GPUs (not able to set different batch_size for train/validate)
Train and validate on different GPUs (can set different batch_size)
Train on all GPUs and save the model per epoch, later run the model on validation data. (not able to use early stopping on validation loss)

What is the best practice?
Any other thoughts and suggestions will be appreciated.