There are a few different ways to tackle this project
If you have time have a look at how its done here:
  
      

      GitHub
  

  
    

GitHub - CSAILVision/semantic-segmentation-pytorch: Pytorch implementation... 3

  Pytorch implementation for Semantic Segmentation/Scene Parsing on MIT ADE20K dataset - GitHub - CSAILVision/semantic-segmentation-pytorch: Pytorch implementation for Semantic Segmentation/Scene Par...