Spectral Representations for Convolutional Neural Networks
Discrete Fourier transforms provide a significant speedup in the computation of convolutions in deep learning. In this work, we demonstrate that, beyond its advantages for efficient computation, the spectral domain also provides a powerful representation in which to model and train convolutional neural networks (CNNs). We employ spectral representations to introduce a number of innovations to CNN design. First, we propose spectral pooling, which performs dimensionality reduction by truncating the representation in the frequency domain. This approach preserves considerably more information per parameter than other pooling strategies and enables flexibility in the choice of pooling output dimensionality. This representation also enables a new form of stochastic regularization by randomized modification of resolution. We show that these methods achieve competitive results on classification and approximation tasks, without using any dropout or max-pooling. Finally, we demonstrate the effectiveness of complex-coefficient spectral parameterization of convolutional filters. While this leaves the underlying model unchanged, it results in a representation that greatly facilitates optimization. We observe on a variety of popular CNN configurations that this leads to significantly faster convergence during training.
Comments/questions about spectral pooling: - L212: "First [spectral pooling] representation maintains more information for the same number of degrees of freedom."-- I disagree with this statement.The amount of information in an image or its DFT is the same, after all it is just a linear transformation of the data.More information is present because the output of spectral pooling is larger than typical max/avg-pooling. In fact, avg-pooling is very similar to spectral pooling: it's a moving average filter (a different low-pass filter) followed by downsampling.The same amount of info could be kept using max/avg-pooling by using overlapping windows (also related to the claim on L250 that max-pooling has a large reduction of dimensionality). - maxpooling is translation invariant, while spectral pooling is not translation invariant (translation is encoded in the phase of the DFT).Supposedly translation invariance helps in recognition tasks.Does spectral pooling get similar results to max-pooling on object recognition tasks (e.g., ImageNet tested in Fig 4a)? - given the similarity to avg-pooling, it would be nice to see a comparison on CIFAR.Comments/questions about spectral parameterization: - for spectral parameterization, it would be interesting to see whether the re-parameterization allows better learning for smaller training set sizes. - L301 - Does the reparameterization affect the regularization of the weights?I suppose not, since it is based on a unitary linear transform.It is worth mentioning this. - L319 - unitary transform is also a condition for the training to be the same. Otherwise regularization of the weights is affected. - L424-425 - The experiments show that fewer iterations are needed for the spectral representation.But this ignores the time-cost for calculating the DFT and inverse DFT.What is the speed-up in terms of actual clock time?