Video Instance Matting
Conventional video matting outputs one alpha matte for all instances
appearing in a video frame so that individual instances are not distinguished.
While video instance segmentation provides time-consistent instance masks,
results are unsatisfactory for matting applications, especially due to applied
binarization. To remedy this deficiency, we propose Video Instance
Matting~(VIM), that is, estimating alpha mattes of each instance at each frame
of a video sequence. To tackle this challenging problem, we present MSG-VIM, a
Mask Sequence Guided Video Instance Matting neural network, as a novel baseline
model for VIM. MSG-VIM leverages a mixture of mask augmentations to make
predictions robust to inaccurate and inconsistent mask guidance. It
incorporates temporal mask and temporal feature guidance to improve the
temporal consistency of alpha matte predictions. Furthermore, we build a new
benchmark for VIM, called VIM50, which comprises 50 video clips with multiple
human instances as foreground objects. To evaluate performances on the VIM
task, we introduce a suitable metric called Video Instance-aware Matting
Quality~(VIMQ). Our proposed model MSG-VIM sets a strong baseline on the VIM50
benchmark and outperforms existing methods by a large margin. The project is
open-sourced at https://github.com/SHI-Labs/VIM.
654b115d2903299c2c636768
Jiachen Li
654b115d2903299c2c636769
Roberto Henschel
admin_assigned
2023-11-08T09:07:10.464Z
6316145f95b55e2621d0a128
/avatars/6a35dbd6f48b6e34a6fc8b8d04e00f6a.svg
Roberto Henschel
user
robhen
654b115d2903299c2c63676a
Vidit Goel
admin_assigned
2023-11-08T09:07:16.949Z
636c0c1a15cd58e915bb8139
/avatars/7c675ac6a7d303d3425e498c4e939eb0.svg
Vidit Goel
user
vidit98
654b115d2903299c2c63676b
Marianna Ohanyan
654b115d2903299c2c63676c
Shant Navasardyan
admin_assigned
2023-11-08T09:07:28.913Z
641dadf816ef0e5a7275a5cd
https://cdn-avatars.huggingface.co/v1/production/uploads/641dadf816ef0e5a7275a5cd/YrieHABBK9VH82VxYPJnp.jpeg
Shant Navasardyan
user
navasardyanshant
654b115d2903299c2c63676d
Humphrey Shi
admin_assigned
2023-11-08T09:07:36.420Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
Video Instance Matting
Conventional video matting outputs one alpha matte for all instances
appearing in a video frame so that individual instances are not distinguished.
While video instance segmentation provides time-consistent instance masks,
results are unsatisfactory for matting applications, especially due to applied
binarization. To remedy this deficiency, we propose Video Instance
Matting~(VIM), that is, estimating alpha mattes of each instance at each frame
of a video sequence. To tackle this challenging problem, we present MSG-VIM, a
Mask Sequence Guided Video Instance Matting neural network, as a novel baseline
model for VIM. MSG-VIM leverages a mixture of mask augmentations to make
predictions robust to inaccurate and inconsistent mask guidance. It
incorporates temporal mask and temporal feature guidance to improve the
temporal consistency of alpha matte predictions. Furthermore, we build a new
benchmark for VIM, called VIM50, which comprises 50 video clips with multiple
human instances as foreground objects. To evaluate performances on the VIM
task, we introduce a suitable metric called Video Instance-aware Matting
Quality~(VIMQ). Our proposed model MSG-VIM sets a strong baseline on the VIM50
benchmark and outperforms existing methods by a large margin. The project is
open-sourced at https://github.com/SHI-Labs/VIM.
MSG-VIM, a neural network for video instance matting, improves temporal consistency in alpha matte predictions using mask and feature guidance, outperforming existing methods on the VIM50 benchmark.
mask sequence guided alpha mattes video instance matting mask augmentations temporal mask guidance temporal feature guidance Video Instance-aware Matting Quality VIMQ VIM50