In this work, we first conduct mathematical analysis on the memory, which is defined as a function that maps an element in a sequence to the current output, of three RNN cells; namely, the simple recurrent neural network (SRN), the long short-term memory (LSTM) and the gated recurrent unit (GRU). Based on the analysis, we propose a new design, called the extended-long short-term memory (ELSTM), to extend the memory length of a cell. Next, we present a multi-task RNN model that is robust to previous erroneous predictions, called the dependent bidirectional recurrent neural network (DBRNN), for the sequence-in-sequenceout (SISO) problem. Finally, the performance of the DBRNN model with the ELSTM cell is demonstrated by experimental results.
The paper proposes a new recurrent cell and a new way to make predictions for sequence tagging. It starts with a theoretical analysis of memory capabilities in different RNN cells and goes on with experiments on POS tagging and dependency parsing. There are serious presentation issues in the paper, which make it hard to understand the ideas and claims.
First, I was not able to understand the message of the theoretical analysis from Section 2 and could not see how it is different from similar derivations (i.e. using a linearized version of an RNN and eigenvalue decomposition) that can be found in many other papers, including (Bengio et al, 1994) and (Pascanu et al, 2013). Novelty aside, the analysis has presentation issues. SRN is introduced without a nonlinearity from the beginning, although normally it should have one. From the classical upper bound with a power of the largest singular value the paper concludes that “Clearly, the memory will explode if \lambda_{max} > 1”, which is not true: the memory *may* explode, having an exponentially growing upper bound does not mean that it *will* explode. The notation chosen from LSTM is  different from the standard in deep learning community and was very hard to understand (Y_t is used instead of h_t, and h_t is used instead of c_t). This notation also does not seem consistent with the rest of the paper, for example Equations 28 and 29 suggest that Y_t are discrete outputs and not vectors. 
The novel cell SLSTM-I is meant to be different from LSTM by addition of “input weight vector c_i”, but is not explained where c_i come from. Are they trainable vectors, one for each time step? If yes, then how could such a cell be applied to sequence which are longer than the training ones?
Equations 28, 29, 30 describe a very unusual kind of a Bidirectional Recurrent Network. To the best of my knowledge it is much more common to make one prediction based on future and past information, whereas the paper describes an approach in which first predictions are made separately based on the past and on the future. It is also very common to use several BiRNN layers, whereas the paper only uses one. As for the proposed DBRNN method, unfortunately, I was not able to understand it.
I also have concerns regarding the experiments. Why is seq2seq without attention is used? On such small datasets attention is likely to make a big difference. What’s the point of reporting results of an LSTM without output nonlinearity (Table 5)?
To sum up, the paper needs a lot work on many fronts, but most importantly, presentation should be improved.