Unsupervised Learning of Artistic Styles with Archetypal Style Analysis
In this paper, we introduce an unsupervised learning approach to automatically discover, summarize, and manipulate artistic styles from large collections of paintings. Our method is based on archetypal analysis, which is an unsupervised learning technique akin to sparse coding with a geometric interpretation. When applied to neural style representations from a collection of artworks, it learns a dictionary of archetypal styles, which can be easily visualized. After training the model, the style of a new image, which is characterized by local statistics of deep visual features, is approximated by a sparse convex combination of archetypes. This enables us to interpret which archetypal styles are present in the input image, and in which proportion. Finally, our approach allows us to manipulate the coefficients of the latent archetypal decomposition, and achieve various special effects such as style enhancement, transfer, and interpolation between multiple archetypes.
Summary:  The authors introduce an unsupervised approach to learn structure in the space of artistic styles, which allows to represent an arbitrary artistic image as a combination of “basis”  or “archetypal” styles, as well as change the input image (resynthesize it) by modifying a contribution of each of the archetypal styles. Following Gatys et al., an artistic style is defined as a set of first and second order statistics (concatenated in a vector) in layers of a pre-trained deep network (authors use VGG-19). The authors compute such parametric styles representations for a collection of artistic images, apply SVD for dimensionality reduction (to make vectors 4096-dim), and use archetypal analysis on the resulting data set of 4096-dim style representations. Given a data set X, the archetypal analysis method allows to learn a dictionary Z of vectors, allowing to approximately represent each x = \sum{\alpha_i * z_i}, where \alpha_i sum up to one (elements of a simplex). Dictionary vectors can themselves be decomposed z = \sum{\beta_i * x_i} in terms of data vectors as well. This allows to compute corresponding mean features maps and covariance matrices of conv layers for vectors in Z (by linearly combining those of x_i), which makes Z a dictionary of styles.  For synthesis the authors suggest an extension to the method of Li et al., consisting of whitening and color transforms in the feature space of the DNN (using the means and covariance matrices of styles images), and trained encoder / decoder pairs for fast mapping to and from features spaces. The proposed extension add a parameter to explicitly model a trade-off between style and content lacking in the original model.  Overall, for any image the method allows to find its decomposition into archetypal styles, visualize (using the synthesis) each of them as textures, have input image stylized using archetypal styles, and modify the style of the input image by changing the decomposition coefficients and resynthesizing it.  Confidence score:  I’m fairly familiar with the literature and practical implementations of style transfer, and I think I understood the proposed method relatively well.  Quality:  The paper is good quality containing thorough description of the method and quite extensive experimental results.  Clarity:  The paper is clearly written and easy to follow.  Originality:  The paper addresses the problem of decomposing and visualizing artistic styles, combing ideas from style transfer literature (universal style transfer) and unsupervised learning (archetypal analysis). Such analysis of styles is novel as it gives a way to discover structure in the space of styles, while prior work mostly focused on synthesis and not on decompositions of styles.  Significance:  The problem of decomposition of styles, or finding factors of variation in styles, is difficult. There are methods allowing to transfer only certain aspects of the style image (e.g. color, fine- or coarse-grained details, …), but a principled description of what the elementary components of a style are is missing. This submission is a step in this direction. Though the archetypal styles are themselves quite complex and could be difficult to interpret (it is not that each archetypal texture is either a color blob or a repeated brush stroke, or something else basic enough to form a “linearly independent basis“ in the space of styles), but nevertheless they provide a new way to describe styles, and the proposed methods allows for fine-grained modifications of input images, which is interesting for applications. Therefore I believe this submission should be accepted for publication.