I think I fixed that issue. I didn’t seem to be passing the right weights from the PS to the workers.
for k, v in self.model.state_dict().items():
    self.conn.tensorset(k, v.cpu().numpy())

Which is what I wrote above, but I actually had v.data.cpu().numpy() in my code.
Now my problem is that my model gives empty predicts.
I am distributing a UNet for image segmentation. I’m utilizing gradient accumulation as a tradeoff for the framework performance using a redisai db. Even though the weights and gradients are being passed properly now, my model is decreasing it’s loss… but also accuracy (IoU) md resulting in empty predictions.
It might be the gradient accumulation and batch normalization or it could be the ReLU activations on the decoding half of the model.
Unless someone knows how UNets nuances perform over a distributed framework?