@pritamdamania87,
First of all, thank you for your reply.

If I understand your comment correctly, It seems I both had training with even and uneven batch size across all GPUs.
The log shows 2931 which is the uneven number of batches assigned for a single GPU. however, in another training session, a even number of batches (like 1466) have also experienced the same issue.
Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?
I will check to join API try to get around this issue of course.

Thank you.!