Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention
Hayeon Kim, Eun Cheol Lee, Yongseok Seo, Donghyuck Im, In Kwon Lee
Automatic identification of fictional characters is one of the primary analysis techniques for video content. A common approach to detect characters in live-action movies involves detecting human faces; however, this approach cannot be used in non-realistic domains, such as animation movies. Detection of characters in animation movies presents two major challenges: the same subject of character can be expressed in various unique styles and there are no stylistic or other restrictions on the nature and design of character objects. To address these challenges, we introduce the "animation adaptive region-based convolutional neural network (AA-R-CNN)" model to detect characters in animation movies and determine whether the detected characters are human or non-human types. Our model extends the Faster R-CNN model, which is a two-stage object detector, in the following manner: 1) we add a hierarchical animation adaptation (HAA) module to learn the variety of unique styles from animation movies using a single model; 2) we incorporate a double-detector architecture to focus on the regions that are visually important in determining the character class. We build a new dataset for the animated character detection task. Experiments on this dataset show that our model outperforms other existing representative object detector models in terms of character detection. Furthermore, our model achieves significant performance improvements compared with previous state-of-the-art methods used for the character dictionary generation task. Our model is robust for a variety of animation styles and can find common visual representations of all types of characters, providing an effective way to detect animated characters.
IEEE Transactions on Multimedia
https://doi.org/10.1109/TMM.2020.3006372
10.1109/TMM.2020.3006372
Fingerprint Dive into the research topics of 'Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention'. Together they form a unique fingerprint.
Animation Engineering & Materials Science
Detectors Engineering & Materials Science
Glossaries Engineering & Materials Science
Neural networks Engineering & Materials Science
Kim, H., Lee, E. C., Seo, Y., Im, D., & Lee, I. K. (Accepted/In press). Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention. IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2020.3006372
Kim, Hayeon ; Lee, Eun Cheol ; Seo, Yongseok ; Im, Donghyuck ; Lee, In Kwon. / Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention. In: IEEE Transactions on Multimedia. 2020.
@article{e3f48ac2f28c45d092cb71b31d2c070b,
title = "Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention",
abstract = "Automatic identification of fictional characters is one of the primary analysis techniques for video content. A common approach to detect characters in live-action movies involves detecting human faces; however, this approach cannot be used in non-realistic domains, such as animation movies. Detection of characters in animation movies presents two major challenges: the same subject of character can be expressed in various unique styles and there are no stylistic or other restrictions on the nature and design of character objects. To address these challenges, we introduce the "animation adaptive region-based convolutional neural network (AA-R-CNN)" model to detect characters in animation movies and determine whether the detected characters are human or non-human types. Our model extends the Faster R-CNN model, which is a two-stage object detector, in the following manner: 1) we add a hierarchical animation adaptation (HAA) module to learn the variety of unique styles from animation movies using a single model; 2) we incorporate a double-detector architecture to focus on the regions that are visually important in determining the character class. We build a new dataset for the animated character detection task. Experiments on this dataset show that our model outperforms other existing representative object detector models in terms of character detection. Furthermore, our model achieves significant performance improvements compared with previous state-of-the-art methods used for the character dictionary generation task. Our model is robust for a variety of animation styles and can find common visual representations of all types of characters, providing an effective way to detect animated characters.",
author = "Hayeon Kim and Lee, {Eun Cheol} and Yongseok Seo and Donghyuck Im and Lee, {In Kwon}",
note = "Publisher Copyright: IEEE Copyright: Copyright 2020 Elsevier B.V., All rights reserved.",
doi = "10.1109/TMM.2020.3006372",
journal = "IEEE Transactions on Multimedia",
Kim, H, Lee, EC, Seo, Y, Im, D & Lee, IK 2020, 'Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention', IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2020.3006372
Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention. / Kim, Hayeon; Lee, Eun Cheol; Seo, Yongseok; Im, Donghyuck; Lee, In Kwon.
In: IEEE Transactions on Multimedia, 2020.
T1 - Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention
AU - Kim, Hayeon
AU - Lee, Eun Cheol
AU - Seo, Yongseok
AU - Im, Donghyuck
AU - Lee, In Kwon
N1 - Publisher Copyright: IEEE Copyright: Copyright 2020 Elsevier B.V., All rights reserved.
N2 - Automatic identification of fictional characters is one of the primary analysis techniques for video content. A common approach to detect characters in live-action movies involves detecting human faces; however, this approach cannot be used in non-realistic domains, such as animation movies. Detection of characters in animation movies presents two major challenges: the same subject of character can be expressed in various unique styles and there are no stylistic or other restrictions on the nature and design of character objects. To address these challenges, we introduce the "animation adaptive region-based convolutional neural network (AA-R-CNN)" model to detect characters in animation movies and determine whether the detected characters are human or non-human types. Our model extends the Faster R-CNN model, which is a two-stage object detector, in the following manner: 1) we add a hierarchical animation adaptation (HAA) module to learn the variety of unique styles from animation movies using a single model; 2) we incorporate a double-detector architecture to focus on the regions that are visually important in determining the character class. We build a new dataset for the animated character detection task. Experiments on this dataset show that our model outperforms other existing representative object detector models in terms of character detection. Furthermore, our model achieves significant performance improvements compared with previous state-of-the-art methods used for the character dictionary generation task. Our model is robust for a variety of animation styles and can find common visual representations of all types of characters, providing an effective way to detect animated characters.
AB - Automatic identification of fictional characters is one of the primary analysis techniques for video content. A common approach to detect characters in live-action movies involves detecting human faces; however, this approach cannot be used in non-realistic domains, such as animation movies. Detection of characters in animation movies presents two major challenges: the same subject of character can be expressed in various unique styles and there are no stylistic or other restrictions on the nature and design of character objects. To address these challenges, we introduce the "animation adaptive region-based convolutional neural network (AA-R-CNN)" model to detect characters in animation movies and determine whether the detected characters are human or non-human types. Our model extends the Faster R-CNN model, which is a two-stage object detector, in the following manner: 1) we add a hierarchical animation adaptation (HAA) module to learn the variety of unique styles from animation movies using a single model; 2) we incorporate a double-detector architecture to focus on the regions that are visually important in determining the character class. We build a new dataset for the animated character detection task. Experiments on this dataset show that our model outperforms other existing representative object detector models in terms of character detection. Furthermore, our model achieves significant performance improvements compared with previous state-of-the-art methods used for the character dictionary generation task. Our model is robust for a variety of animation styles and can find common visual representations of all types of characters, providing an effective way to detect animated characters.
U2 - 10.1109/TMM.2020.3006372
DO - 10.1109/TMM.2020.3006372
JO - IEEE Transactions on Multimedia
JF - IEEE Transactions on Multimedia
Kim H, Lee EC, Seo Y, Im D, Lee IK. Character Detection in Animation Movies Using Multi-Style Adaptation and Visual Attention. IEEE Transactions on Multimedia. 2020. https://doi.org/10.1109/TMM.2020.3006372