Fashion Compatibility Modeling through a Multi modal Try on guided Scheme
Recent years have witnessed a growing trend of fashion compatibility modeling, which scores the matching degree of the given outfit and then provides people with some dressing advice. Existing methods have primarily solved this problem by analyzing the discrete interaction among multiple complementary items. However, the fashion items would present certain occlusion and deformation when they are worn on the body. Therefore, the discrete item interaction cannot capture the fashion compatibility in a combined manner due to the neglect of a crucial factor: the overall try on appearance. In light of this, we propose a multi modal try on guided compatibility modeling scheme to jointly characterize the discrete interaction and try on appearance of the outfit. In particular, we first propose a multi modal try on template generator to automatically generate a try on template from the visual and textual information of the outfit, depicting the overall look of its composing fashion items. Then, we introduce a new compatibility modeling scheme which integrates the outfit try on appearance into the traditional discrete item interaction modeling. To fulfill the proposal, we construct a large scale real world dataset from SSENSE, named FOTOS, consisting of 11,000 well matched outfits and their corresponding realistic try on images. Extensive experiments have demonstrated its superiority to state of the arts.
SIGIR