yes, the output of g^(1) is the hidden state.
the arrow on the top feeds the hidden state to the next layer.
For the arrow on the right, it feeds the hidden state and cell state (if cell is lstm cell); it feeds the hidden state (if cell is gru cell).
u can read this 1