Is this behavior reproducible?
Do you only encounter it in a distributed setup or also with a single GPU?
Could you post a code snippet to reproduce this issue and post some information about your current system and setup?
CC @yikaiw