For other people want to look at my issue, let implement the first point of ptrblck

currently you are using  lossSeg  to calculate gradients in both models,  GenA  and  SegA

So the code will be
fake_A= GenA(real_A)
lossGen=L1(real_A, fake_A)
# Feed fake_A to SegA
pred_A= SegA(fake_A.detach())
lossSeg = cross_entropy(pred_A, label)
loss = lossSeg + lossGen
loss.backward()

Is it correct?