title: PolyMaX: General Dense Prediction with Mask Transformer
summary: Dense prediction tasks, such as semantic segmentation, depth estimation, and
  surface normal prediction, can be easily formulated as per-pixel classification
  (discrete outputs) or regression (continuous outputs). This per-pixel
  prediction paradigm has remained popular due to the prevalence of fully
  convolutional networks. However, on the recent frontier of segmentation task,
  the community has been witnessing a shift of paradigm from per-pixel prediction
  to cluster-prediction with the emergence of transformer architectures,
  particularly the mask transformers, which directly predicts a label for a mask
  instead of a pixel. Despite this shift, methods based on the per-pixel
  prediction paradigm still dominate the benchmarks on the other dense prediction
  tasks that require continuous outputs, such as depth estimation and surface
  normal prediction. Motivated by the success of DORN and AdaBins in depth
  estimation, achieved by discretizing the continuous output space, we propose to
  generalize the cluster-prediction based method to general dense prediction
  tasks. This allows us to unify dense prediction tasks with the mask transformer
  framework. Remarkably, the resulting model PolyMaX demonstrates
  state-of-the-art performance on three benchmarks of NYUD-v2 dataset. We hope
  our simple yet effective design can inspire more research on exploiting mask
  transformers for more dense prediction tasks. Code and model will be made
  available.
paper_authors._id: 65519e3d45c0dccd57473ef3
paper_authors.name: Xuan Yang
paper_authors._id: 65519e3d45c0dccd57473ef4
paper_authors.name: Liangzhe Yuan
paper_authors._id: 65519e3d45c0dccd57473ef5
paper_authors.name: Kimberly Wilber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T10:34:13.539Z
paper_authors._id: 630d6b268e3ff0c723322bff
paper_authors.avatarUrl: /avatars/af0a07c2fb49f9e93e897631de0ba0a7.svg
paper_authors.fullname: Kimberly Wilber
paper_authors.type: user
paper_authors.user: kwilber
paper_authors._id: 65519e3d45c0dccd57473ef6
paper_authors.name: Astuti Sharma
paper_authors._id: 65519e3d45c0dccd57473ef7
paper_authors.name: Xiuye Gu
paper_authors._id: 65519e3d45c0dccd57473ef8
paper_authors.name: Siyuan Qiao
paper_authors._id: 65519e3d45c0dccd57473ef9
paper_authors.name: Stephanie Debats
paper_authors._id: 65519e3d45c0dccd57473efa
paper_authors.name: Huisheng Wang
paper_authors._id: 65519e3d45c0dccd57473efb
paper_authors.name: Hartwig Adam
paper_authors._id: 65519e3d45c0dccd57473efc
paper_authors.name: Mikhail Sirotenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T16:29:52.533Z
paper_authors._id: 65b13272dfa3789ef39937ce
paper_authors.avatarUrl: /avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg
paper_authors.fullname: Mikhail Sirotenko
paper_authors.type: user
paper_authors.user: msirotenko-g
paper_authors._id: 65519e3d45c0dccd57473efd
paper_authors.name: Liang-Chieh Chen
paper_title: PolyMaX: General Dense Prediction with Mask Transformer
paper_summary: Dense prediction tasks, such as semantic segmentation, depth estimation, and
  surface normal prediction, can be easily formulated as per-pixel classification
  (discrete outputs) or regression (continuous outputs). This per-pixel
  prediction paradigm has remained popular due to the prevalence of fully
  convolutional networks. However, on the recent frontier of segmentation task,
  the community has been witnessing a shift of paradigm from per-pixel prediction
  to cluster-prediction with the emergence of transformer architectures,
  particularly the mask transformers, which directly predicts a label for a mask
  instead of a pixel. Despite this shift, methods based on the per-pixel
  prediction paradigm still dominate the benchmarks on the other dense prediction
  tasks that require continuous outputs, such as depth estimation and surface
  normal prediction. Motivated by the success of DORN and AdaBins in depth
  estimation, achieved by discretizing the continuous output space, we propose to
  generalize the cluster-prediction based method to general dense prediction
  tasks. This allows us to unify dense prediction tasks with the mask transformer
  framework. Remarkably, the resulting model PolyMaX demonstrates
  state-of-the-art performance on three benchmarks of NYUD-v2 dataset. We hope
  our simple yet effective design can inspire more research on exploiting mask
  transformers for more dense prediction tasks. Code and model will be made
  available.
paper_ai_summary: A cluster-prediction method using mask transformers is generalized to achieve state-of-the-art performance across various dense prediction tasks, including depth estimation and surface normal prediction.
paper_ai_keywords: transformer architectures mask transformers cluster-prediction per-pixel prediction depth estimation surface normal prediction DORN AdaBins continuous output space discrete output space PolyMaX NYUD-v2 dataset