@shivammehta007 Can you try with find_unused_parameters=True? Also, can you provide a self contained repro of the issue?
Another option is you could use set_static_graph: pytorch/distributed.py at master · pytorch/pytorch · GitHub 10, if the parameters used in each iteration of your model are always the same.