@albanD that is not what I was expecting, because gru1_opt.step() would help the netwrok predict better in the next time instant following step(). Is there any workaround? Also I got the following to work:
for i in range(100):
    gru1_opt.zero_grad()
    gru2_opt.zero_grad()
    vector = torch.randn((15, 100, 2))
    gru1_output, _ = gru1(vector)  # (15, 100, 200)
    gru2_output, _ = gru2(gru1_output)  # (15, 100, 2)
    loss_gru1 = criterion(gru2_output, torch.randn((15, 100, 2)))
    loss_gru1.backward(retain_graph=True)
    gru1_opt.step()
    gru1_output, _ = gru1(vector)
    gru2_output, _ = gru2(gru1_output)  # (15, 100, 2)
    loss_gru2 = criterion(gru2_output, torch.randn((15, 100, 2)))
    loss_gru2.backward(retain_graph=True)
    gru2_opt.step()
    print(f"GRU1 loss: {loss_gru1.item()}, GRU2 loss: {loss_gru2.item()}")

But I guess I will run into memory issues.
PS: In the first backprop I only need update the weights of GRU1. In the second backprop, I need to update the weights of GRU2
In essence, I want to flow the gradients through GRU2 and update the weights of GRU1 in the first instance. In the second part I need to update the weights of GRU2 only.