Hi @Sayak_Paul, thanks for sharing the links!
The problem is at inference time, and sure there are a lot of good documentation like the TensorFlow Distributed Training or the Keras ones that you linked above, but all of these demonstrate how to make use of multiple GPUs at training time.
One of the things that I tried was to create a @tf.function and use the run method of the MirroredStrategy, but this didn’t work out. The issue is that I was getting lots of Nones for reasons I couldn’t understand.
What the StackOverflow solution points out is the use of the tf.gather method, that joins the output data back along the first dimension, as Laplace Ricky points out.
As far as I’m concerned, this is the first solution that I could find at inference time.