Hi,
Since last week I upgraded pytorch to 1.10.0, this error occurred. Here’s the log:

Blockquote
WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15342 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 15343 closing signal SIGHUP
Traceback (most recent call last):
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File “/home/anaconda3/envs/aa/lib/python3.8/runpy.py”, line 87, in _run_code
exec(code, run_globals)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 193, in 
main()
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 189, in main
launch(args)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launch.py”, line 174, in launch
run(args)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/run.py”, line 710, in run
elastic_launch(
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 252, in launch_agent
result = agent.run()
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py”, line 125, in wrapper
result = f(*args, **kwargs)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 709, in run
result = self._invoke_run(role)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py”, line 843, in _invoke_run
time.sleep(monitor_interval)
File “/home/anaconda3/envs/aa/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py”, line 60, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 15265 got signal: 1