Published as a conference paper at ICLR 2018 DEEP COMPLEX NETWORKS
At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their realvalued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.
This paper defines building blocks for complex-valued convolutional neural networks: complex convolutions, complex batch normalisation, several variants of the ReLU nonlinearity for complex inputs, and an initialisation strategy. The writing is clear, concise and easy to follow.
An important argument in favour of using complex-valued networks is said to be the propagation of phase information. However, I feel that the observation that CReLU works best out of the 3 proposed alternatives contradicts this somewhat. CReLU simply applies ReLU component-wise to the real and imaginary parts, which has an effect on the phase information that is hard to conceptualise. It definitely does not preserve phase, like modReLU would.
This makes me wonder whether the "complex numbers" paradigm is applied meaningfully here, or whether this is just an arbitrary way of doing some parameter sharing in convnets that happens to work reasonably well (note that even completely random parameter tying can work well, as shown in "Compressing neural networks with the hashing trick" by Chen et al.). Some more insight into how phase information is used, what it represents and how it is propagated through the network would help to make sense of this.
The image recognition results are mostly inconclusive, which makes it hard to assess the benefit of this approach. The improved performance on the audio tasks seems significant, but how the complex nature of the networks helps achieve this is not really demonstrated. It is unclear how the phase information in the input waveform is transformed into the phase of the complex activations in the network (because I think it is implied that this is what happens). This connection is a bit vague. Once again, a more in-depth analysis of this phase behavior would be very welcome.
I'm on the fence about this work: I like the ideas and they are explained well, but I'm missing some insight into why and how all of this is actually helping to improve performance (especially w.r.t. how phase information is used).
Comments:
- The related work section is comprehensive but a bit unstructured, with each new paragraph seemingly describing a completely different type of work. Maybe some subsection titles would help make it feel a bit more cohesive.
- page 3: "(cite a couple of them)" should be replaced by some actual references :)
- Although care is taken to ensure that the complex and real-valued networks that are compared in the experiments have roughly the same number of parameters, doesn't the complex version always require more computation on account of there being more filters in each layer? It would be nice to discuss computational cost as well.
REVISION: I have decided to raise my rating from 5 to 7 as I feel that the authors have adequately addressed many of my comments. In particular, I really appreciated the additional appendix sections to clarify what actually happens as the phase information is propagated through the network.
Regarding the CIFAR results, I may have read over it, but I think it would be good to state even more clearly that these experiments constitute a sanity check, as both reviewer 1 and myself were seemingly unaware of this. With this in mind, it is of course completely fine that the results are not better than for real-valued networks.