  Prevalent semantic segmentation solutions are, in essence, a dense
discriminative classifier of p(class|pixel feature). Though straightforward,
this de facto paradigm neglects the underlying data distribution p(pixel
feature|class), and struggles to identify out-of-distribution data. Going
beyond this, we propose GMMSeg, a new family of segmentation models that rely
on a dense generative classifier for the joint distribution p(pixel
feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs)
via Expectation-Maximization (EM), so as to capture class-conditional
densities. Meanwhile, the deep dense representation is end-to-end trained in a
discriminative manner, i.e., maximizing p(class|pixel feature). This endows
GMMSeg with the strengths of both generative and discriminative models. With a
variety of segmentation architectures and backbones, GMMSeg outperforms the
discriminative counterparts on three closed-set datasets. More impressively,
without any modification, GMMSeg even performs well on open-world datasets. We
believe this work brings fundamental insights into the related fields.

Computer Vision and Pattern Recognition