KERNEL RNN LEARNING (KERNL)
We describe Kernel RNN Learning (KeRNL), a reduced-rank, temporal eligibility trace-based approximation to backpropagation through time (BPTT) for training recurrent neural networks (RNNs) that gives competitive performance to BPTT on long time-dependence tasks. The approximation replaces a rank-4 gradient learning tensor, which describes how past hidden unit activations affect the current state, by a simple reduced-rank product of a sensitivity weight and a temporal eligibility trace. In this structured approximation motivated by node perturbation, the sensitivity weights and eligibility kernel time scales are themselves learned by applying perturbations. The rule represents another step toward biologically plausible or neurally inspired ML, with lower complexity in terms of relaxed architectural requirements (no symmetric return weights), a smaller memory demand (no unfolding and storage of states over time), and a shorter feedback time. Finally, we show that KeRNL can learn long time-scales more efficiently than BPTT in an online setting.
This paper proposes a simple method for performing temporal credit assignment in RNN training.  While it seems somewhat naive and unlikely to work (in my opinion), the experimental results surprisingly show reasonable performance on several reasonably challenging artificial tasks.
The core of the approach is based on equation 7, which approximates the Jacobian between different hidden states at different time-steps as a single adaptively-learned matrix times a decay factor that depends on the time gap.  While this seems like a very severe approximation to make the authors speculate that some kind of feedback alignment-like mechanism might be at play.
The presentation needs work in several areas, and the experimental results require more explanation, but otherwise this seems like a solid paper.  I would probably increase my rating if the authors could address my issues satisfactorily. 
See below for more detailed comments:
Abstract & Section 1: 
Is "sensitivity tensor" or "credit assignment tensor" common term?  Because I've never heard them before.  Consider defining them before you discuss it, and using consistent jargon.  Later in Section 2 you seem to call this the "RTRL tensor" (whose meaning I can infer).  
Section 2:
Gradient vanishing isn't so much a problem in itself, but a symptom that the sensitivity of the network's output to the action of some neuron in the past is very low. Ths gradient is just relaying this information, so I don't really see vanishing gradients as the problem to overcome, but rather low sensitivity on past activations.
Section 3: 
Did you mean to write (W^out h^t + b^out) instead of (W^out h + b^out)^t ?
"[equation] represents the gradient of the cost with respect to the current hidden state".  The RHS of this equation makes no sense to me.  Not only does this not depend on the nonlinearity in any way, it doesn't include any consideration of future outputs on which the current h surely depends. 
It would make the paper much more pleasant to read if you gave your derivation of the learning rule before you stated it in gory detail.  It feels almost completely arbitrary reading it first without any justification. This might be fine if it were compact and elegant, but it's not.
Consider using exp(x) instead of e^x since the symbol e already means something else in your notation.
Section 4:
Please define "temporal variation"
Section 5:
You should elaborate on the experimental setup you used.  Especially for the Addition and MNIST problems.  For example, what consistutes a "step" in figure 2?  Does KeRL take "one" step per time-step?  Or does "step" mean a complete gradient computation from running from t = 1 to t= T?  Is the BPTT truncated?  Are you counting one step of BPTT to be one complete forwards and backwards pass?
You should include some basic description of what an IRNN is.
When you say that for MNIST that KeRL "does not converge to as good of an optimum" this seems like unjustified inference.  You don't really know that it is converging to a minimum of the original objective at all.  It could be converging to the minimum of some other objective it is implicitly optimizing due to your approximations (if one even exists).  Or it could be simply cycling around and failing to converge.  The fact that the loss plateaus isn't direct evidence of convergence in any sense.  If you wanted to measure this more directly you could look at the (true) gradient magnitude.
"only requires a few tensor operations at each time step" -> this is also true of UORO