I’m still unsure which issue we are debugging, but in any case:
color_loss = criterion(Avg_color,Target_color_match)

won’t calculate the gradients for the model parameters, since you were recreating tensors and are thus detaching the computation graph in:
Avg_color = [torch.tensor(x,dtype=float,requires_grad=False) for x in styleGAN_transformedimage]