I’m testing out parallelizing across multiple worker nodes in a parameter-server type architecture. I’m using redisAI to handle the model weight and gradient sharing between primary and worker nodes. At each node the worker retains it’s own parameter set and delivers gradients to the primary node, which updates the global model. I have four workers, so I combine each workers update together
Sequentially performing worker updates, global update, and then worker reads of updated global model is a performance hit. I’m also testing out a update-and-continue scheme where the worker will push it’s gradients to the global model and then continue with it’s own path instead of adjusting to the global model.
beta = 0.25
gmsd = model.state_dict()
for name, param in model.named_parameters():
            worker_001_data = redisai_conn.tensorget(f'worker_001:{name}_grad')
            worker_002_data = redisai_conn.tensorget(f'worker_002:{name}_grad')
            worker_003_data = redisai_conn.tensorget(f'worker_003:{name}_grad')
            worker_004_data = redisai_conn.tensorget(f'worker_004:{name}_grad')
            tens = worker_001_data*beta + worker_002_data*beta + worker_003_data*beta + worker_004_data*beta
            worker_ten = torch.from_numpy(tens).to(self.device)
            if gmsd[name].grad == None:
                gmsd[name].grad = (worker_ten)
            else:
                gmsd[name].grad.copy_(worker_ten)

model.load_state_dict(gmsd)

My goal is improved reliability in the global models predictions.