I think the problem lie here in the optimizer parameters , I did this line as you recommended in another post
parameters = list(model_in.parameters()) + list(model_RNN.parameters()) + list(model_out.parameters())

optimizer = torch.optim.SGD(parameters, lr=learning_rate)