Distributed UNet over Parameter Server Output Empty vision
  
  
    I have distributed a custom UNet over a parameter server (PS) (over a network and not locally distributed). The workers receive weights, compute the gradient and push it back to the PS. 
I am accumulating gradients on workers to minimize network usage. After a few gradient pushes, the UNet starts outputting empty predicting as loss decreases but so does accuracy (testing the framework with interval model testing). 
import torch.nn as nn
import torch
import torch.nn.functional as F
import torch.q…
  


Link to that question for anyone in the future’s use.