Thanks ptrblck. Yes. I only want to use grad of lossSeg to compute the parameter of SegA. So, as your suggestion, my code will be
fake_A= GenA(real_A)
lossGen=L1(real_A, fake_A)
lossGen.backward()

# Feed fake_A to SegA
pred_A= SegA(fake_A.detach())
lossSeg = cross_entropy(pred_A, label)
lossSeg.backward()

Is it correct?