oh… That’s what I was looking for!
Since receiving the your answer, I have been googled quite a bit, but there is not much content. um…
Is there any function that supports apply gradient? like apply_gradients()
    a_grads = autograd.grad(a_loss, tm.a.parameters(), retain_graph=True) 
    for layer, p in enumerate(tm.a.parameters()):
        p.grad = torch.tensor(a_grads[layer])

    b_grads = autograd.grad(b_loss, tm.b.parameters()) 
    for layer, p in enumerate(tm.b.parameters()):
        p.grad = torch.tensor(a_grads[layer])

    tm.a_optimizer.step()  
    tm.b_optimizer.step()

anyway I confirmed that this seems to be learning. Thank you for your answers so far.