googlebot:

I have trouble understanding your intentions in this code. In particular, h_out is gradient, but it is handled the same way as regular output y_out.


Yes! That is exactly my intention. I want to implement a neural net which outputs not only the value, but also the gradient of another neural net. Thus the gradient is computed in the forward pass.
That is what I stated in my initial post. Apparently, this forum does not render latex-code, so here I try it again, maybe it is more readable.
Given an input x, and a hidden state h, the output of the RNN should be computed as
y = F(x,h)
The next hidden state should be computed as
h_next = dF/dh(x, h)
Here F is a multilayer perceptron that I want to train.
The training data consists of sequences (x_t,y_t)