I can confirm this is indeed a bug. Please track the progress of the fix: [torch/elastic] Scale down does not work correctly when agent is killed with SIGINT, SIGTERM · Issue #67742 · pytorch/pytorch · GitHub 6. The fix itself is quite simple.