FINAL UPDATE: I think I found the source of the problem - it’s the .resize_as_(). So, if instead of
self.TotalWGrad = torch.zeros(1).resize_as_(self.Weights)
I have the following:
self.TotalWGrad = torch.zeros(self.NumEdges)
then the gradients are consistent.