Turn out to be a stupid mistake by me.
embed_pool = torch.load(os.path.join(args.saved_plugin_dir, 'embed_pool.pth'))

should be changed to
embed_pool = torch.load(os.path.join(args.saved_plugin_dir, 'embed_pool.pth'), map_location=torch.device('cpu'))

torch.load() will automatically map the file to device:0, and this device is the same in the eye of each device, thus causing the problem (spawning additional processes in device:0).
Mark this problem as solved by myself.