Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?


You would probably have to add some logging code to your application which records the number of batches on each rank as part of the training loop.