Thanks @lcw you are right looks like v1.9 uses a lot more file descriptors (fds) compared to v1.7.
While v1.7 was close to 135 fds/process, it was close to 270 fds/process for version 1.9, with the zero rank process having 960 fds. These numbers are for 3 GPU and 5 CPU processes.
Yes, increasing the ulimit fixed the issue for v1.9, which points to another possible improvement
In v1.7 we have a very clear error message for such a scenario, the error message looks like
OSError: [Errno 24] Too many open files: '/home/user/anaconda3/envs/pytorch1.7/lib/python3.8/multiprocessing/queues.py'

Probably such an error message for v1.9 and future versions will be useful