@ptrblck In my original network, I take raw audio time series that has the shape (ch, h, w) = (64, 1, 151) just before the permute operation. After permute, the shape is (1, 64, 151). Thus I can treat it as image data and use 2D kernels in the conv layers. The network I provided as an example is the example of a full precision (original) network. It works great.
When I am converting it to XNOR like this:
conv → bn → relu ->maxpool
bn → BinConv2d → relu → maxpool
.
.
.
Now, my forward function looks like this:
def forward(self, x):
        for m in self.modules():
            if isinstance(m, nn.BatchNorm2d):
                if hasattr(m.weight, 'data'):
                    m.weight.data.clamp_(min=0.01)
        x = self.block1(x);
        x = x.permute((0, 2, 1, 3));
        x = self.self.block2(x);
        y = self.output[0](x);
        return y;

I have BinActive function:
class BinActive(torch.autograd.Function):
    '''
    Binarize the input activations and calculate the mean across channel dimension.
    '''
    @staticmethod
    def forward(self, input):
        self.save_for_backward(input)
        # size = input.size()
        input = input.sign()
        return input

    @staticmethod
    def backward(self, grad_output):
        input, = self.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input.ge(1)] = 0
        grad_input[input.le(-1)] = 0
        return grad_input

class BinConv2d(nn.Module): # change the name of BinConv2d
    def __init__(self, ch_in, ch_out, kernel_size=1, stride=1, padding=0, bias=False):
        super(BinConv2d, self).__init__()
        self.ch_in = ch_in
        self.ch_out = ch_out
        self.layer_type = 'BinConv2d'
        self.kernel_size = kernel_size
        self.stride = stride
        self.padding = padding
        self.bias = bias;

        self.conv = nn.Conv2d(self.ch_in, self.ch_out, kernel_size=self.kernel_size, stride=self.stride, padding=self.padding, bias=False)
        # nn.init.kaiming_normal_(self.conv.weight, nonlinearity='relu'); # kaiming with relu is equivalent to he_normal in keras

    def forward(self, x):
        x = BinActive.apply(x)
        x = self.conv(x)
        return x

I hope this clarifies your question. Please let me know if you need some more information