Skimming through your code it looks like the view on your target tensors might be wrong. Since your image channels are in dim2 when loading the image, you should use image.permute(2, 0, 1) to shift the channels to dim0.
Also, could you explain your code a bit?
It seems you are feeding the a and b image planes from the lab format to your models.
As far as I understand, you would like to use grayscale images as inputs, i.e. the l plane, and predict a and b.