Complex Gated Recurrent Neural Networks
Complex numbers have long been favoured for digital signal processing, yet complex representations rarely appear in deep learning architectures. RNNs, widely used to process time series and sequence information, could greatly benefit from complex representations. We present a novel complex gated recurrent cell, which is a hybrid cell combining complex-valued and norm-preserving state transitions with a gating mechanism. The resulting RNN exhibits excellent stability and convergence properties and performs competitively on the synthetic memory and adding task, as well as on the real-world tasks of human motion prediction.
Complex-valued representations have begun receiving some attention in the the deep learning community but they have been applied only to the most basic of architectures [1,10,36]. For recurrent networks, complex representations could gain more acceptance if they were shown to be compatible with more commonly used gated architectures and also competitive for real-world data. This is exactly the aim of this work, where we propose a complex-valued gated recurrent network and show how it can easily be implemented with a standard deep learning library such as TensorFlow. Our contributions can be summarized as follows 2 :
• We introduce a novel complex-gated recurrent unit; to the best of our knowledge, we are the first to explore such a structure using complex number representations.
• We compare experimentally the effects of a bounded versus unbounded non-linearity in recurrent networks, finding additional evidence countering the commonly held heuristic that only bounded non-linearities should be applied in RNNs. In our case unbounded non-linearities perform better, but must be coupled with the stabilizing measure of using norm-preserving state transition matrices.
• Our complex gated network is stable and fast to train; it outperforms the state of the art with equal parameters on synthetic tasks and delivers state-of-the-art performance one the real-world application of predicting poses in human motion capture using fewer weights.
Summary of approach and contributions: The authors resurrect the pioneering work of Hirose on complex valued neural networks in order to provide a new RNN based on a complex valued activation/transition function and a complex argument gating mechanism. In order to obtain a differentiable function that is not constant and yet bounded, the authors step away from holomorphic functions and employ CR calculus. The authors show experimental improvements on two synthetic tasks and one actual data set.  Strengths of the paper: o) Moving away from strict holomorphy and using CR calculus to apply complex valued networks to RNNs is interesting as a novel technique. I think that the authors should spend more time explaining how phases can be easily encoded in the complex domain and therefore why such complex representations can be advantageous for sequential learning. o) The paper is cleanly written and the architectural changes simple enough that they can be written in a few lines of a standard deep learning package such as tensorflow. o) The study of Hirose vs modReLU and the impact of learning transition matrices on the Stiefel Manifold is interesting as it gives additional evidence helping compare two radically different approaches to maintaining stability in a RNN. Such a study fits nicely within recent architectural developments in the field of RNNs.  Weaknesses of the paper: o) The authors rely on CR calculus to differentiate the complex functions they employ and, more importantly, design an architecture where key mechanisms (equation (7) and (13)) could have a similar construction with real numbers and additional numbers of dimensions. For instance (13) is analogous to a bilinear gating mechanism which has been employed studied in https://arxiv.org/pdf/1705.03122.pdf for sequential learning. I believe that in order to isolate the effect of learning complex valued values it would be necessary to compare the performance of the proposed architecture with a real valued competitor network that would employ the same family of non-linearities, in particular bi-linear gates as in (13). o) The authors provide very few insights as to why a complex valued representation could be better for sequential learning. I believe that even simple elements of signal processing could expose in a paragraph theoretical elements that would help consider the source of experimental improvement as coming from a complex valued representation. o) The experiments themselves are not most convincing. It seems surprising that no neural translation or neural language understanding benchmark is featured in the paper. The only real-work task should feature the size of the networks involved in numbers of parameters and MB as well as a comparison of computational complexity. Without such elements, it is not certain that better results could be achieved by a larger real valued network.  Overall the paper appears original and well written but could really benefit from more substantial theoretical and experimental insights designed to isolate the role of complex values in the network from non-linearity design changes.  The author's feedback has better convinced me that complex values play a fundamental positive role in the architecture. I indeed am familiar with the ease of use of complex representations and their expressiveness in the field of time series and the addition of sequential tasks will help that point clearer. In that respect, I appreciate that the authors will add new experimental results to make their experimental claim more robust with TIMIT and WMT data. I revised my rating accordingly.