thelastspark:

Furthermore, since his computer would connect and disconnect, it would need someway of getting the latest model parameters


Does this mean you would need elasticity (allow worker processes to join/leave dynamically) in torch.distributed.rpc?