I don’t quite understand, how the target_label approach works. It seems you are creating a new model and optimizer for each label and assume that this model would then predict only the current target class vs. the rest?
If so, it might work, but could you post an executable code snippet (using random tensors) to reproduce the initial issue?