title: Towards Natural Image Matting in the Wild via Real-Scenario Prior
summary: Recent approaches attempt to adapt powerful interactive segmentation models,
  such as SAM, to interactive matting and fine-tune the models based on synthetic
  matting datasets. However, models trained on synthetic data fail to generalize
  to complex and occlusion scenes. We address this challenge by proposing a new
  matting dataset based on the COCO dataset, namely COCO-Matting. Specifically,
  the construction of our COCO-Matting includes accessory fusion and
  mask-to-matte, which selects real-world complex images from COCO and converts
  semantic segmentation masks to matting labels. The built COCO-Matting comprises
  an extensive collection of 38,251 human instance-level alpha mattes in complex
  natural scenarios. Furthermore, existing SAM-based matting methods extract
  intermediate features and masks from a frozen SAM and only train a lightweight
  matting decoder by end-to-end matting losses, which do not fully exploit the
  potential of the pre-trained SAM. Thus, we propose SEMat which revamps the
  network architecture and training objectives. For network architecture, the
  proposed feature-aligned transformer learns to extract fine-grained edge and
  transparency features. The proposed matte-aligned decoder aims to segment
  matting-specific objects and convert coarse masks into high-precision mattes.
  For training objectives, the proposed regularization and trimap loss aim to
  retain the prior from the pre-trained model and push the matting logits
  extracted from the mask decoder to contain trimap-based semantic information.
  Extensive experiments across seven diverse datasets demonstrate the superior
  performance of our method, proving its efficacy in interactive natural image
  matting. We open-source our code, models, and dataset at
  https://github.com/XiaRho/SEMat.
paper_authors._id: 670f125e97bbe79a5028fe6c
paper_authors.name: Ruihao Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:00.931Z
paper_authors._id: 65896f5bae21a8ff2883ec1a
paper_authors.avatarUrl: /avatars/ffacc166fd0d145504773c666b7c8b94.svg
paper_authors.fullname: Ruihao Xia
paper_authors.type: user
paper_authors.user: XiaRho
paper_authors._id: 670f125e97bbe79a5028fe6d
paper_authors.name: Yu Liang
paper_authors._id: 670f125e97bbe79a5028fe6e
paper_authors.name: Peng-Tao Jiang
paper_authors._id: 670f125e97bbe79a5028fe6f
paper_authors.name: Hao Zhang
paper_authors._id: 670f125e97bbe79a5028fe70
paper_authors.name: Qianru Sun
paper_authors._id: 670f125e97bbe79a5028fe71
paper_authors.name: Yang Tang
paper_authors._id: 670f125e97bbe79a5028fe72
paper_authors.name: Bo Li
paper_authors._id: 670f125e97bbe79a5028fe73
paper_authors.name: Pan Zhou
paper_title: Towards Natural Image Matting in the Wild via Real-Scenario Prior
paper_summary: Recent approaches attempt to adapt powerful interactive segmentation models,
  such as SAM, to interactive matting and fine-tune the models based on synthetic
  matting datasets. However, models trained on synthetic data fail to generalize
  to complex and occlusion scenes. We address this challenge by proposing a new
  matting dataset based on the COCO dataset, namely COCO-Matting. Specifically,
  the construction of our COCO-Matting includes accessory fusion and
  mask-to-matte, which selects real-world complex images from COCO and converts
  semantic segmentation masks to matting labels. The built COCO-Matting comprises
  an extensive collection of 38,251 human instance-level alpha mattes in complex
  natural scenarios. Furthermore, existing SAM-based matting methods extract
  intermediate features and masks from a frozen SAM and only train a lightweight
  matting decoder by end-to-end matting losses, which do not fully exploit the
  potential of the pre-trained SAM. Thus, we propose SEMat which revamps the
  network architecture and training objectives. For network architecture, the
  proposed feature-aligned transformer learns to extract fine-grained edge and
  transparency features. The proposed matte-aligned decoder aims to segment
  matting-specific objects and convert coarse masks into high-precision mattes.
  For training objectives, the proposed regularization and trimap loss aim to
  retain the prior from the pre-trained model and push the matting logits
  extracted from the mask decoder to contain trimap-based semantic information.
  Extensive experiments across seven diverse datasets demonstrate the superior
  performance of our method, proving its efficacy in interactive natural image
  matting. We open-source our code, models, and dataset at
  https://github.com/XiaRho/SEMat.
paper_ai_summary: A new dataset, COCO-Matting, and a model architecture, SEMat, are introduced to enhance the performance of interactive matting in complex natural scenes by leveraging fine-grained features and trimap-based training objectives.
paper_ai_keywords: SAM interactive matting COCO dataset COCO-Matting accessory fusion mask-to-matte feature-aligned transformer matte-aligned decoder regularization trimap loss