Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from O(n 2 ) to O(n log n). These non-parametric and low computational properties construct extremely efficient networks in terms of the number of parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 49.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.
This paper presents a new pointwise convolution (PC) method which applies conventional transforms such as DWHT and DCT. The proposed method aims to reduce the computational complexity of CNNs without degrading the performance. Compared with the original PC layer, the DWHT/DCT-based methods do not require any learnable parameters and reduce the floating-point operations. The paper also empirically optimizes the networks by removing ReLU after the proposed PC layers and using conventional transforms for high-level features extraction. Experiments on CIFAR100 show that the DWHT-based model improves the accuracy and reduces parameters and FLOPs compared with MobileNet-V1.
Although this paper is well organized and easy to follow, the novelty of the proposal seems limited and the performance improvement claimed by the author(s) is not very convincing due to the insufficiency of experiments. Firstly, the proposed method is just a manually designed and fixed 1*1 convolutional kernel, and its superiority over random initialization seems very limited as shown in Table 1. Also, the proposed method makes accuracy degrade when applied to low- and middle-level features. I wonder whether there is a more theoretical explanation for that. Moreover, the experiments are performed only on a small dataset CIFAR100. According to my own experience, the artificial convolutional kernels with some prior knowledge may work well on small datasets but tend to fail on larger ones. More experiments on larger-scale datasets like ImageNet are recommended to make results more convincing.
Therefore, my decision leans to a rejection.
Some questions:
1. How is the performance when applying RCPC only to low-/middle-/high-level features? I suppose it should be proved that the proposed method is definitely better than random initialization.
2. Why is only applying the proposed block to high-level layers working? How is the trade-off between parameters and accuracy different for each level of features?
Spelling mistake:
Page 6: in the second last paragraph, 'non-parameteric' should be 'non-parametric'.