I think you are trying to implement the image-level features. The image-level features are exploited in ParseNet and it is implemented by a global average pooling. You can confirm it (section 3.3), but I almost sure about that. I will start to do the same work applied to COCO2017 dataset for semantic segmentation.