Definition: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
Input: A major challenge in the training of recurrent neural networks is the so-called vanishing or exploding gradient problem. The use of a norm-preserving transition operator can address this issue, but parametrization is challenging. In this work we focus on unitary operators and describe a parametrization using the Lie algebra u(n) associated with the Lie group U(n) of n × n unitary matrices. The exponential map provides a correspondence between these spaces, and allows us to define a unitary matrix using n real coefficients relative to a basis of the Lie algebra. The parametrization is closed under additive updates of these coefficients, and thus provides a simple space in which to do gradient descent. We demonstrate the effectiveness of this parametrization on the problem of learning arbitrary unitary operators, comparing to several baselines and outperforming a recently-proposed lower-dimensional parametrization. We additionally use our parametrization to generalize a recently-proposed unitary recurrent neural network to arbitrary unitary matrices, using it to solve standard long-memory tasks. Introduction While recurrent neural networks (RNNs) are seeing widespread success across many tasks, the fundamental architecture presents challenges to typical training algorithms. In particular, the problem of ‘vanishing/exploding gradients’ (Hochreiter 1991) in gradient-based optimization persists, where gradients either vanish or diverge as one goes deeper into the network, resulting in slow training or numerical instability. The long short-term memory (LSTM) network (Hochreiter and Schmidhuber 1997) was designed to overcome this issue. Recently, the use of norm-preserving operators in the transition matrix the matrix of weights connecting subsequent internal states of the RNN have been explored (Arjovsky, Shah, and Bengio 2016; Mikolov et al. 2015; Le, Jaitly, and Hinton 2015). Using operators with bounded eigenvalue spectrum should, as demonstrated by Arjovsky, Shah, and Bengio (2016), bound the norms of the gradients in the network, assuming an appropriate nonlinearity is applied. Unitary matrices satisfy this requirement and are the focus of this work. Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Imposing unitarity (or orthogonality) on this transition matrix is however challenging for gradient-based optimization methods, as additive updates typically do not preserve unitarity. Solutions include re-unitarizing after each batch, or using a parametrization of unitary matrices closed under addition. In this work we propose a solution in the second category, using results from the theory of Lie algebras and Lie groups to define a general parametrization of unitary matrices in terms of skew-Hermitian matrices (elements of the Lie algebra associated to the Lie group of unitary matrices). As explained in more detail below, elements of this Lie algebra can be identified with unitary matrices, while the algebra is closed under addition, forming a vector space over real numbers. While we are motivated by the issues of RNNs, and we consider an application in RNNs, our primary focus here is on a core question: how can unitary matrices be learned? Assuming the choice of a unitary transition matrix is an appropriate modelling choice, the gradients on this operator should ultimately guide it towards unitarity, if it is possible under the parametrization or learning scheme used. It is therefore useful to know which approach is best. We distil the problem into its simplest form (a learning task described in detail later), so that our findings cannot be confounded by other factors specific to the RNN long-term memory task, before demonstrating our parametrization in that setting. Related work We draw most inspiration from the recent work of Arjovsky, Shah, and Bengio (2016), who proposed a specific parametrization of unitary matrices and demonstrated its utility in standard long-term memory tasks for RNNs. We describe their parametrization here, as we use it later. Citing the difficulty of obtaining a general and efficient parametrization of unitary matrices, they use the fact that the unitary group is closed under matrix multiplication to form a composite operator: U = D3R2FD2ΠR1FD1 (1) where each component is unitary and easily parametrized: • D is a diagonal matrix with entries of the form e, α ∈ R • R is a complex reflection operator; R = I − 2 vv † ‖v‖2 († denotes Hermitian conjugate) ar X iv :1 60 7. 04 90 3v 3 [ st at .M L ] 1 0 Ja n 20 17 • F andF−1 are the Fourier and inverse Fourier transforms (or, in practice, their discrete matrix representations) • Π is a fixed permutation matrix In total, this parametrization has 7n real learnable parameters (2n for each reflection and n for each diagonal operator), so describes a subspace of unitary matrices (which have n real parameters). Nonetheless, they find that an RNN using this operator as its transition matrix outperforms LSTMs on the adding and memory tasks described first in Hochreiter and Schmidhuber (1997). This prompted us to consider other parametrizations of unitary matrices which might be more expressive or interpretable. Mikolov et al. (2015) constrain a part of the transition matrix to be close to the identity, acting as a form of long-term memory store, while Le, Jaitly, and Hinton (2015) initialize it to the identity, and then use ReLUs as non-linearities. Henaff, Szlam, and LeCun (2016) study analytic solutions to the long-term memory task, supporting observations and intuitions that orthogonal (or unitary) matrices would be appropriate as transition matrices for this task. They also study initializations to orthogonal and identity matrices, and consider experiments where an additional term in the loss function encourages an orthogonal solution to the transition matrix, without using an explicit parametrization. Saxe, McClelland, and Ganguli (2014) study exact solutions to learning dynamics in deep networks and find that orthogonal weight initializations at each layer lead to depth-independent learning (thus escaping the vanishing/exploding gradient problem). Interestingly, they attribute this to the eigenvalue spectrum of orthogonal matrices lying on the unit circle. They compare with weights initialized to random, scaled Gaussian values, which preserve norms in expectation (over values of the random matrix) and find orthogonal matrices superior. It therefore appears that preserving norms is not sufficient to stabilize gradients over network depth, but that the eigenvalue spectrum must also be strictly controlled. In a related but separate vein, Krueger and Memisevic (2016) penalize the difference of difference of norms between subsequent hidden states in the network. This is not equivalent to imposing orthogonality of the transition matrix, as the norm of the hidden state may be influenced by the inputs and non-linearities, and their method directly addresses this norm. The theory of Lie groups and Lie algebras has seen most application in machine learning for its use in capturing notions of invariance. For example, Miao and Rao (2007), learn infinitesimal Lie group generators (elements of the Lie algebra) associated with affine transformations of images, corresponding to visual perceptual invariances. This is different to our setting as our generators are already known (we assume the Lie group U(n)) and wish to learn the coefficients of a given transformation relative to that basis set of generators. However, our approach could be extended to the case where the basis of u(n) is unknown, and must be learned. As we find later (appendix B), the choice of basis can impact performance, and so may be an important consideration. Cohen and Welling (2014) learn commutative subgroups of SO(n) (known as toroidal subgroups), motivated by learning the irreducible representations of the symmetry group corresponding to invariant properties of images. Their choice of group parametrization is equivalent to selecting a particular basis of the corresponding Lie algebra, as they describe, but primarily exploit the algebra to understand properties of toroidal subgroups. Tuzel, Porikli, and Meer (2008) perform motion estimation by defining a regression function in terms of a function on the Lie algebra of affine transformations, and then learning this. This is similar to our approach in the sense that they do optimization in the Lie algebra, although as they consider two-dimensional affine transformations only, their parametrization of the Lie algebra is straight forward. Finally, Hazan, Kale, and Warmuth (2016) describe an online learning algorithm for orthogonal matrices – which are the real-valued equivalent to unitary matrices. They also claim that the approach is extends easily to unitary matrices. Structure of this paper We begin with an introduction to the relevant facts and definitions from the theory of Lie groups and Lie algebras, to properly situate this work in its mathematical context. Further exposition is beyond the scope of this paper, and we refer the interested reader to any of the comprehensive introductory texts on the matter. We explain our parametrization in detail and describe a method to calculate the derivative of the matrix exponential a quantity otherwise computationally intractable. Then, we describe a simple but clear experiment designed to test our core question of learning unitary matrices. We compare to an approach using the parametrization of Arjovsky, Shah, and Bengio (2016) and one using polar decomposition to ‘backproject’ to the closest unitary matrix. We use this experimental set-up to probe aspects of our model, studying the importance of the choice of basis (appendix B), and the impact of the restricted parameter set used by one of the alternate approaches. We additionally implement our parametrization in a recurrent neural network as a ‘general unitary RNN’, and evaluate its performance on standard long-memory tasks. The Lie algebra u(n) Basics of Lie groups and Lie algebras A Lie group is a group which is also a differentiable manifold, with elements of the group corresponding to points on the manifold. The group operations (multiplication and inversion) must be smooth maps (infinitely differentiable) back to the group. In this work we consider the group U(n): the set of n × n unitary matrices, with matrix multiplication. These are the complex-valued analogue to orthogonal matrices, satisfying the property U†U = UU† = I (2) where † denotes the conjugate transpose (or Hermitian conjugate). Unitary matrices preserve matrix norms, and have eigenvalues lying on the (complex) unit circle, which is the desired property of the transition matrix in a RNN. The differentiable manifold property of Lie groups opens the door for the study of the Lie algebra. This object is the tangent space to the Lie group at the identity (the group must<lb>have an identity element). Consider a curve through the Lie<lb>group U(n) a one-dimensional subspace parametrized by<lb>a variable t, where U(t = 0) = I (this is a matrix U(t) in<lb>U(n) parametrised by t, not a group). Consider the defin-<lb>ing property of unitary matrices (Equation 2), and take the<lb>derivative along this curve:<lb>U(t)†U(t) = I→ U̇(t)†U(t) + U†(t)U̇(t) = 0 (3)<lb>Taking t→ 0, U(t)→ I, we have<lb>U̇(0)†I + I†U̇(0) = 0⇒ U̇(0)† = −U̇(0) (4)<lb>The elements U̇(0) belong to the Lie algebra. We refer to<lb>this Lie algebra as u(n), and an arbitrary element as L. Then<lb>Equation 4 defines the properites of these Lie algebra ele-<lb>ments; they are n× n skew-Hermitian matrices: L† = −L.<lb>As vector spaces, Lie algebras are closed under addition.<lb>In particular u(n) is a vector space over R, so a real linear<lb>combination of its elements is once again in u(n) (this is also<lb>clear from the definition of skew-Hermitian). We exploit this<lb>fact later.<lb>Lie algebras are also endowed with an operation known as<lb>the Lie bracket, which has many interesting properties, but is<lb>beyond the scope of this work. Lie algebras are interesting<lb>algebraic objects and have been studied deeply, but in this<lb>work we use u(n) because of the exponential map.<lb>Above, it was shown that elements of the algebra can be<lb>derived from the group (considering infinitesimal steps away<lb>from the identity). There is a reverse operation, allowing el-<lb>ements of the group to be recovered from the algebra: this is<lb>the exponential map. In the case of matrix groups, the expo-<lb>nential map is simply the matrix exponential:<lb>exp(L) =<lb>∞<lb>∑ j=0<lb>L<lb>j!<lb>(5)<lb>Very simply, L ∈ u(n), then exp(L) ∈ U(n). While this<lb>map is not in general surjective, it so happens that U(n) is<lb>a compact, connected group and so exp is indeed surjective<lb>(Tao 2011). That is, for any U ∈ U(n), there exists some<lb>L ∈ u(n) such that exp(L) = U . Notably, while orthogonal<lb>matrices also form a Lie group O(n), with associated Lie<lb>algebra o(n) consisting of skew-symmetric matrices, O(n)<lb>is not connected, and so the exponential map can only pro-<lb>duce special orthogonal matrices those with determinant<lb>one SO(n) being the component of O(n) containing the<lb>identity.<lb>Parametrization of U(n) in terms of u(n)<lb>The dimension of u(n) as a real vector space is n. This is<lb>readily derived from noting that an arbitrary n× n complex<lb>matrix has 2n free real parameters, and the requirement of<lb>L† = −L imposes n constraints. So, a set of n linearly-<lb>independent skew-Hermitian matrices defines a basis for the<lb>space;<lb>{Tj}j={1,...,n2}. Then any element L can be written<lb>as<lb>L =<lb>n<lb>∑ j=1<lb>λjTj<lb>(6)<lb>where<lb>{λj}j=1,...,n2 are n real numbers; the coefficients of<lb>L with respect to the basis. Using the exponential map,<lb>U = exp(L) = exp  n<lb>∑ j=1<lb>λjTj <lb>(7)<lb>we see that these<lb>{λj}j=1,...,n2 suffice as parameters of U<lb>(given the basis Tj). This is the parametrization we propose.<lb>It has two attractive properties:<lb>1. It is a fully general parametrization, as the exponential<lb>map is surjective<lb>2. Gradient updates on<lb>{λj}j=1,...,n2 preserve unitarity au-<lb>tomatically, as the algebra is closed under addition<lb>This parametrization means gradient steps are taken in the<lb>vector space of u(n), rather than the manifold of U(n),<lb>which may provide a flatter cost landscape although con-<lb>firming this intuition would require further analysis. This<lb>work is intended to explore the use of this parametrization<lb>for learning arbitrary unitary matrices.<lb>There are many possible choices of basis for u(n). We<lb>went for the following set of sparse matrices:<lb>1. n diagonal, imaginary matrices: Ta is i on the a-th diago-<lb>nal, else zero.<lb>2. n(n−1)<lb>2 symmetric, imaginary matrices with two non-zero<lb>elements, e.g., for n = 2,<lb>(<lb>0 i<lb>i 0<lb>)<lb>3. n(n−1)<lb>2 anti-symmetric, real matrices with two non-zero<lb>elements, e.g., for n = 2,<lb>(<lb>0 1<lb>−1 0<lb>) We explore the effects of choice of basis in appendix B.<lb>Derivatives of the matrix exponential<lb>The matrix exponential appearing in Equation 7 poses an is-<lb>sue for gradient calculations. In general, the derivative of the<lb>matrix exponential does not have a closed-form expression,<lb>so computing gradients is intractable.<lb>In early stages of this work, we used the method of finite<lb>differences to approximate gradients, which would prohibit<lb>its use in larger-scale applications (such as RNNs). In the ap-<lb>pendix we describe an investigation into using random pro-<lb>jections to overcome this limitation, which while promising<lb>turned out to yield minimal benefit.<lb>We therefore sought mathematical solutions to this com-<lb>plexity issue, which we describe here and in further detail in<lb>the appendix. Exploiting the fact that L is skew-Hermitian,<lb>we can derive an analytical expression for the derivative of<lb>U with respect to each of its parameters, negating the need<lb>for finite differences.<lb>This expression takes the form:<lb>∂U<lb>∂λa<lb>= WVaW †<lb>(8)<lb>where W is a unitary matrix of eigenvectors obtained in<lb>the eigenvalue decomposition of U ; U = WDW †, (D =<lb>diag(d1, . . . , dn2 ); di are the eigenvalues of U ). Each Va is a matrix defined component-wise<lb>i = j :Vii = (W<lb>TaW )iie di<lb>(9)<lb>i 6= j :Vij = (W<lb>TaW )ij<lb>(<lb>ei − ej<lb>di − dj<lb>)<lb>(10)<lb>Where Ta is the basis matrix of the Lie algebra in the a-th<lb>direction.<lb>We provide the derivation, based on work<lb>from Kalbfleisch and Lawless (1985) and Jennrich and<lb>Bright (1976) in Appendix A.<lb>We can simplify the expression W<lb>TaW for each Ta, de-<lb>pending on the type of basis element. In these expressions,<lb>wa refers to the a-th row of W.<lb>1. Ta purely imaginary; W<lb>TaW = i · outer(w∗<lb>a,wa)<lb>2. Ta symmetric imaginary, nonzero in positions (r, s) and<lb>(s, r): W<lb>TrsW = i · (outer(w∗<lb>s ,wr) + outer(w∗<lb>r ,ws))<lb>3. Ta antisymmetric real, nonzero in positions (r, s) and<lb>(s, r): W<lb>TrsW = outer(w∗<lb>r ,ws)− outer(w∗<lb>s ,wr)<lb>These expressions follow from the sparsity of the basis and<lb>are derived in appendix A. Thus, we reduce the calculation<lb>of W<lb>TaW from two matrix multplications to at most two<lb>vector outer products.<lb>Overall, we have reduced the cost of calculating gradi-<lb>ents to a single eigenvalue decomposition, and for each pa-<lb>rameter two matrix multiplications (equation 8), one or two<lb>vector outer products, and element-wise multiplication of<lb>two matrices (equations 9, 10). As we see in the RNN ex-<lb>periments, this actually makes our approach faster than the<lb>(restricted)uRNN of (Arjovsky, Shah, and Bengio 2016) for<lb>roughly equivalent numbers of parameters.<lb>Supervised Learning of Unitary Operators<lb>We consider the supervised learning problem of learning the<lb>unitary matrix U that generated a y from x; y = Ux, given<lb>examples of such xs and ys. This is the core learning prob-<lb>lem that needs to be solved for the state-transformation ma-<lb>trix in RNNs. It is similar to the setting considered in Hazan,<lb>Kale, and Warmuth (2016) (they consider an online learning<lb>problem). We compare a number of methods for learning U<lb>at different values of n. We further consider the case where<lb>we have artificially restricted the number of learnable vari-<lb>ables in our parametrization (for the sake of comparison),<lb>and generate a pathological change of basis to demonstrate<lb>the relevance of selecting a good basis (appendix B).<lb>Task<lb>The experimental setup is as follows: we create a n×n uni-<lb>tary matrix U (the next section describes how this is done),<lb>then sample vectors x ∈ C with normally-distributed coef-<lb>ficients. We create yj = Uxj + j where ∼ N (0, σ). The<lb>objective is to recover U from the<lb>{xj ,yj} pairs by min-<lb>imizing the squared Euclidean distance between predicted<lb>and true y values;<lb>U = argmin<lb>U<lb>1<lb>N<lb>N<lb>∑ j<lb>‖ŷj−yj‖ = argmin<lb>U<lb>1<lb>N<lb>N<lb>∑ j<lb>‖Uxj−yj‖<lb>(11)<lb>While this problem is easily solved in the batch setting us-<lb>ing least-squares, we wish to learn U through mini-batch<lb>stochastic gradient descent, to emulate a deep learning sce-<lb>nario.<lb>For each experimental run (a single U ), we generate one<lb>million training<lb>{xj ,yj} pairs, divided into batches of size<lb>20. The test and validation sets both contain 100, 000 ex-<lb>amples. In practice we set σ = 0.01 and use a fixed<lb>learning rate of 0.001. For larger dimensions, we run the<lb>model through the data for multiple epochs, shuffling and<lb>re-batching each time.<lb>All experiments were implemented<lb>in Python. The code is available here:<lb>https://github.com/ratschlab/uRNN. For<lb>the matrix exponential, we use the scipy builtin expm,<lb>which uses Pade approximation (Al-Mohy and Higham<lb>2009). We make use of the fact that iL is Hermitian to use<lb>eigh (also in scipy) to perform eigenvalue decompositions.<lb>Generating the ground-truth unitary matrix<lb>The U we wish to recover is generated by one of three meth-<lb>ods:<lb>1. QR decomposition: we create a n × n complex matrix<lb>with normally-distributed entries and then perform a QR<lb>decomposition, producing a unitary matrix U and an up-<lb>per triangular matrix (which is discarded). This approach<lb>is also used to sample orthogonal matrices in Hazan, Kale,<lb>and Warmuth (2016), noting a result from Stewart (1980)<lb>demonstrating that this is equivalent to sampling from the<lb>appropriate Haar measure.<lb>2. Lie algebra: given the standard basis of u(n), we sam-<lb>ple n normally-distributed real λj to produce U =<lb>exp<lb>(∑<lb>j λjTj<lb>)<lb>3. Unitary composition: we compose parametrized unitary<lb>operators as in Arjovsky, Shah, and Bengio (2016) (Equa-<lb>tion 1). The parameters are sampled as follows: angles in<lb>D come from U(−π, π). The complex reflection vectors<lb>in R come from U(−s, s) where s =<lb>√<lb>6<lb>2n .<lb>We study the effects of this generation method on test-set<lb>loss in a later section. While we find no significant associ-<lb>ation between generation method and learning approach, in<lb>our experiments we nonetheless average over an equal num-<lb>ber of experiments using each method, to compensate for<lb>possible unseen bias.<lb>Approaches<lb>We compare the following approaches for learning U :<lb>1. projection: U is represented as an unconstrained<lb>n × n complex matrix, but after each gradient update we<lb>project it to the closest unitary matrix, using polar decom-<lb>position (Keller 1975). This amounts to 2n real parame-<lb>ters.<lb>2. arjovsky: U is parametrized as in Equation 1, which<lb>comes to 7n real parameters. 3. lie algebra: (we refer to this as u(n)) U is<lb>parametrized by its n real coefficients<lb>{λj} in the Lie<lb>algebra, as in Equation 7.<lb>As baselines we use the true matrix U , and a random<lb>unitary matrix UR generated by the same method as U (in<lb>that experimental run).<lb>We also implemented the algorithm described in Hazan,<lb>Kale, and Warmuth (2016) and considered both unitary and<lb>orthogonal learning tasks (our parametrization contains or-<lb>thogonal matrices as a special case) but found it too numer-<lb>ically unstable and therefore excluded it from our analyses.<lb>Comparison of Approaches<lb>Table 1 shows the test-set loss for different values of n and<lb>different approaches for learning U . We performed between<lb>6 and 18 replicates of each experiment, and show bootstrap<lb>estimates of means and standard errors over these replicates.<lb>As we can see, the learning task becomes more challenging<lb>as n increases, but our parametrization (u(n)) consistently<lb>outperforms the other approaches.<lb>Restricting to 7n parameters<lb>As mentioned, arjovsky uses only 7n parameters. To<lb>check if this difference accounts for the differences in<lb>loss observed in Table 1, we ran experiments where we<lb>fixed all but 7n (selected randomly) of the<lb>{λj} in the<lb>lie algebra parametrization. The fixed parameters re-<lb>tained their initial values throughout the experiment. We<lb>observe that, as suspected, restricting to 7n parameters re-<lb>sults in a performance degradation equivalent to that of<lb>arjovsky.<lb>Table 2 shows the results for n = 8, 14, 20. The fact<lb>that the restricted case is consistently within error of the<lb>arjovsky model supports our hypothesis that the differ-<lb>ence in learnable parameters accounts for the difference in<lb>performance. This suggests that generalising the model of<lb>Arjovsky, Shah, and Bengio to allow for n parameters may<lb>result in performance similar to our approach. However, how<lb>to go about such a generalisation is unclear, as a naive ap-<lb>proach would simply use a composition of n operators, and<lb>this would likely become computationally intractable.<lb>Method of generating U<lb>As described, we used three methods to generate the true<lb>U . One of these produces U in the subspace available to<lb>the composition parametrization (Equation 1), so we were<lb>curious to see if this parametrization performed better on<lb>experiments using that method. We were also concerned that<lb>generating U using the Lie algebra parametrization might<lb>make the task too ‘easy’ for our approach, as its random<lb>initialization could lie close to the true solution.<lb>Figure 1 shows box-plots of the distribution of test losses<lb>from these approaches for the three methods, comparing<lb>our approach (u(n)) with that of Arjovsky, Shah, and Ben-<lb>gio (2016), denoted arjovsky. To combine results from<lb>experiments using different values of n, we first scaled test-<lb>set losses by the performance of rand (the random uni-<lb>tary matrix), so the y-axis ranges from 0 (perfect) to 1 (ran-<lb>dom performance). The dotted line denotes the average (over<lb>0.00<lb>0.05<lb>0.10<lb>0.15 0.000<lb>0.001<lb>0.002<lb>0.003 u(n) u(n) (restricted)<lb>method to<lb>generate U<lb>composition<lb>Lie algebra<lb>QR arjovsky u(n)<lb>te<lb>st<lb>lo<lb>ss<lb>(f<lb>ra<lb>ct<lb>io<lb>n<lb>of<lb>ra<lb>nd<lb>om<lb>lo<lb>ss<lb>) learning approach<lb>Figure 1: We ask whether the method used to generate U<lb>influences performance for different approaches to learning<lb>U . Error bars are bootstrap estimates of 95% confidence in-<lb>tervals. To compare across different n’s, we normalise each<lb>loss by the loss of rand for that n, and reporrt fractions.<lb>The dotted line is the true loss, similarly normalised. the<lb>choice of method to generate U does not appear to affect<lb>test-set loss for the different approaches. Right: Finer reso-<lb>lution on the u(n) result in left panel. We also include the<lb>case where we restrict to 7n learnable parameters. methods) of the test-set loss for true, similarly scaled. The<lb>right panel in Figure 1 shows a zoomed-in version of the<lb>u(n) result where the comparison with true is more mean-<lb>ingful, and a comparison with the case where we have re-<lb>stricted to 7n learnable parameters (see earlier).<lb>We do not observe a difference (within error) between the<lb>methods, which is consistent between u(n) and arjovsky.<lb>Our concern that using the Lie algebra to generate U would<lb>make the task ‘too easy’ for u(n) was seemingly unfounded.<lb>Unitary Recurrent Neural Network for Long<lb>Memory Tasks<lb>To demonstrate that our approach is practical for use in deep<lb>learning, we incorporate it into a recurrent neural network to<lb>solve standard long-memory tasks. Specifically, we define a<lb>general unitary RNN with recurrence relation<lb>ht = f (βUht−1 + V xt + b)<lb>(12)<lb>where f is a nonlinearity, β is a free scaling factor, U is our<lb>unitary matrix parametrised as in equation 7, ht is the hidden<lb>state of the RNN and xt is the input data at ‘time-point’ t.<lb>We refer to this as a ‘general unitary RNN’ (guRNN), to<lb>distinguish it from the restricted uRNN of Arjovsky, Shah,<lb>and Bengio (2016).<lb>We use the guRNN on two tasks: the ‘adding problem’<lb>and the ‘memory problem’, first described in (Hochreiter<lb>and Schmidhuber 1997). For the sake of brevity we refer<lb>to (Arjovsky, Shah, and Bengio 2016) for specific exper-<lb>imental details, as we use an identical experimental setup<lb>(reproduced in TensorFlow; see above github link for code).<lb>We compare our model (guRNN) with the restricted uRNN<lb>(ruRNN) parametrised as in equation 1, a LSTM (Hochre-<lb>iter and Schmidhuber 1997), and the IRNN of Le, Jaitly,<lb>and Hinton (2015). Figure 2 shows the results for each n<lb>true<lb>projection<lb>arjovsky<lb>lie algebra<lb>rand<lb>3 6.004± 0.005× 10−4 8 ± 1 6.005± 0.003× 10−4 6.003± 0.003× 10−4 12.5± 0.4<lb>6<lb>∼ 0.001<lb>15± 1<lb>0.09± 0.01<lb>0.03± 0.01<lb>24 ± 1<lb>8<lb>∼ 0.002<lb>14± 1<lb>1.17± 0.06<lb>0.014± 0.006 31.6± 0.6<lb>14<lb>∼ 0.003<lb>24± 4<lb>10.8± 0.3<lb>0.07± 0.02<lb>52± 1<lb>20<lb>∼ 0.004<lb>38± 3<lb>29.0± 0.5<lb>0.47± 0.03<lb>81± 2<lb>Table 1: Loss (mean l2-norm between ŷi and yi) on the test set for the different approaches as the dimension of the unitary<lb>matrix changes. true refers to the matrix used to generate the data, projection is the approach of ‘re-unitarizing’ using<lb>a polar decomposition after gradient updates, arjovsky is the composition approach defined in Equation 1, u(n) is our<lb>parametrization (Equation 7) and rand is a random unitary matrix generated in the same manner as true. Values in bold are<lb>the best for that n (excluding true). The error for true is typically very small, so we omit it.<lb>n<lb>arjovsky lie restricted lie unrestricted<lb>8 1.2± 0.1<lb>1.0± 0.2<lb>0.04± 0.01<lb>14<lb>11.6± 0.3 12.6± 0.4<lb>0.25± 0.03<lb>20 27.8± 0.7 28.0± 0.6<lb>0.19± 0.03<lb>Table 2: We observe that restricting our approach to the same number of learnable parameters as that of (Arjovsky, Shah, and<lb>Bengio 2016) causes a similar degradation in performance on the task. This indicates that the relatively superior performance<lb>of our model is explained by its generality in capturing arbitrary unitary matrices. task where the sequence length or the memory duration is<lb>T = 100.<lb>While our model guarantees unitarity of U , this is not<lb>sufficient to prevent gradients from vanishing. Consider the<lb>norm of the gradient of the cost C with respect to the data at<lb>time τ , and use submultiplicativity of the norm to write;<lb>∥∥∥∥ ∂C<lb>∂xτ ∥∥∥∥ ≤<lb>∥∥∥∥ ∂C<lb>∂xT ∥∥∥∥ (<lb>T−1<lb>∏ t=τ<lb>‖f ′ (Uht + V xt + b) ‖‖U‖<lb>∥∥∥∥∂hτ<lb>xτ ∥∥∥∥<lb>where f ′ is a diagonal matrix giving the derivatives of the<lb>nonlinearity. Using a unitary matrix fixes ‖U‖ = 1, but be-<lb>yond further restrictions (on V and b) does nothing to con-<lb>trol the norm of f ′, which is at most 1 for common nonlin-<lb>earities. Designing a nonlinearity to better preserve gradient<lb>norms is beyond the scope of this work, so we simply scaled<lb>U by a constant multiplicative factor β to counteract the ten-<lb>dency of the nonlinearity to shrink gradients. In Figure 2 we<lb>denote this setup by guRNNβ . Confirming our intuition, this<lb>simple modification greatly improves performance on both<lb>tasks.<lb>Perhaps owing to our efficient gradient calculation (ap-<lb>pendix A) and simpler recurrence relation, our model runs<lb>faster than that of (Arjovsky, Shah, and Bengio 2016) (in our<lb>implementation), by a factor of 4.8 and 2.6 in the adding and<lb>memory tasks shown in Figure 2 respectively. This amounts<lb>to the guRNN processing 61.2 and 37.0 examples per second<lb>in the two tasks, on a GeForce GTX 1080 GPU.<lb>Discussion<lb>Drawing from the rich theory of Lie groups and Lie algebras,<lb>we have described a parametrization of unitary matrices ap-<lb>propriate for use in deep learning. This parametrization ex-<lb>ploits the Lie group-Lie algebra correspondence through the<lb>exponential map to represent unitary matrices in terms of<lb>real coefficients relative to a given basis of the Lie alge-<lb>bra u(n). As this map from u(n) to U(n) is surjective, the<lb>parametrization can describe any unitary matrix.<lb>We have demonstrated that unitary matrices can be<lb>learned with high accuracy using simple gradient de-<lb>scent, and that this approach outperforms a recently-<lb>proposed parametrization (from Arjovsky, Shah, and Ben-<lb>gio (2016)) and significantly outperforms the approach of<lb>‘re-unitarizing’ after gradient updates. This experimental de-<lb>sign is quite simple, designed to probe a core problem, be-<lb>fore considering the broader setting of RNNs.<lb>Our experiments with general unitary RNNs using this<lb>parametrization showed that this approach is practical for<lb>deep learning. With a fraction of the parameters, our model<lb>outperforms LSTMs on the standard ‘memory problem’ and<lb>attains comparable (although inferior) performance on the<lb>adding problem (Hochreiter and Schmidhuber 1997). Fur-<lb>ther work is required to understand the difference in perfor-<lb>mance between our approach and the ruRNN of (Arjovsky,<lb>Shah, and Bengio 2016) perhaps the 7n-dimensional sub-<lb>space captured by their parametrization is serendipitously<lb>beneficial for these RNN tasks although we note that the<lb>results presented here are not the fruit of exhaustive hyperpa-<lb>rameter exploration. Of particular interest is the impressive<lb>performance of both uRNNs on the memory task, where the<lb>LSTM and IRNN appear to fail to learn.<lb>While our RNN experiments have demonstrated the util-<lb>ity of using a unitary operator for these tasks, we believe<lb>that the role of the nonlinearity in the vanishing and ex-<lb>ploding gradient problem must not be discounted. We have<lb>shown that a simple scaling factor can help reduce the van-<lb>ishing gradient problem induced by the choice of nonlinear-<lb>ity. More analysis considering the combination of nonlinear-<lb>ity and transition operator must be performed to better tackle<lb>this problem.<lb>The success of our parametrization for unitary operator<lb>learning suggests that the approach of performing gradient

Learning Unitary Operators with Help From u(n)