Have you tried to reproduce the results training a simpler 3 class Unet like:

  
      

      keras.io
  

  
    

Keras documentation: Image segmentation with a U-Net-like architecture 2



  

  
    
    
  

  


Check also:
  
      

      keras.io
  

  
    

Keras documentation: Multiclass semantic segmentation using DeepLabV3+ 1



  

  
    
    
  

  



  
      

      Medium – 5 Mar 19
  

  
    

A Keras Pipeline for Image Segmentation 1

  There are hundreds of tutorials on the web which walk you through using Keras for your image segmentation tasks. These are extremely…

  
    Reading time: 9 min read