Apparently, I was mistaken, the problem is something else entirely, although I’m not sure what. There is a square operation in the network that causes the output to go to infinity. The code looks like this:


def forward(self, x):
    x_rel = self.conv1(x)
    x_rel = self.bn1(x_rel)
    x_rel = x_rel ** 2
    x_rel = self.cc(x_rel)
    x_rel = self.bn2(x_rel)
    x_rel = F.relu(x_rel)
    x_app = self.conv2(x)
    x_app = self.bn3(x_app)
    x_app = F.relu(x_app)
    out = torch.cat((x_rel, x_app), dim=1)
    out = self.reduction(out)
    return out

If I comment the square operation out, no problem would occur, but since I’m trying to implement the network proposed in a paper, that would make my implementation unfaithful. If I leave the square operation as is, and comment the validation loop out, nothing will happen either. But if I leave both the square operation and the validation loop, the output of the validation loop will go to infinity, and every training loop after that will also produce infinity output.