Thank you for the help!
Actually, the 1x1 convolutions are in the self.weighted_sum layer in LBCBlock. I’ve tried to follow your advise changing the implementation of this block from
        self.conv_filter = nn.Conv2d(n_channels, n_kernels, kernel_size=3, padding=1, bias=False)
        kernels = torch.tensor(new_kernel(n_channels, n_kernels, sparsity)).type('torch.FloatTensor')
        kernels.requires_grad_()
        self.conv_filter.weight = nn.Parameter(kernels)
        self.conv_filter.weight.detach_()

        self.weighted_sum = nn.Conv2d(n_kernels, n_channels, kernel_size=1)

to
        with torch.no_grad():
            self.conv_filter = nn.Conv2d(n_channels, n_kernels, kernel_size=3, padding=1, bias=False)
            kernels = torch.tensor(new_kernel(n_channels, n_kernels, sparsity)).type('torch.FloatTensor')
            # kernels.requires_grad_()
            self.conv_filter.weight = nn.Parameter(kernels)
            # self.conv_filter.weight.detach_()

        self.weighted_sum = nn.Conv2d(n_kernels, n_channels, kernel_size=1)

and also decreasing the batch size from 64 to 16. Nevertheless, the same error still happens, wich I understand as my model is still too large.
I keep thinking that something is not working the way I expect it to be, since one of the main purpose of using a LBCNN is to have a model with fewer learnable parameters. (If you’re interested: [1608.06049] Local Binary Convolutional Neural Networks)
Even so, thank you again!