Just to make sure I understand, you are trying to predict 12 continuous valued masks from an RGB image?  To me that’s a pretty intense objective, and you might just be running up against your irreducible error.  I’d try the whole setup and see if the model is generalizing to validation data first. You might need a deeper U-net or wider decoding layers.