To isolate the issue a bit more, the first block after permute is: BatchNorm -> BinaryActivation -> Conv2d ->ReLU. All the elements of the output tensor of this Conv2d are zeros. I have checked the output of the BinActivation. They are not all -1’s, there are 1s as well. I have initialised the conv using self.conv.apply(lambda m: nn.init.kaiming_normal_(m.weight, nonlinearity='relu')); which ensures that the initial weights are not zeros.
Any idea what could go wrong? This is only happening for the Binary conv after the permute, not before the permute.