it does not look like you have used self.hidden0 in your forward computation. It’s not clear how it will learn self.hidden0 unless you are supplying it as an argument in forward, but why not just do: output, hidden = self.gru(output, self.hidden0)?