Recent advances in machine learning have brought the field closer to computational creativity research. From a creativity research point of view, this offers the potential to study creativity in relationship with knowledge acquisition. From a machine learning perspective, however, several aspects of creativity need to be better defined to allow the machine learning community to develop and test hypotheses in a systematic way. We propose an actionable definition of creativity as the generation of out-of-distribution novelty. We assess several metrics designed for evaluating the quality of generative models on this new task. We also propose a new experimental setup. Inspired by the usual held-out validation, we hold out entire classes for evaluating the generative potential of models. The goal of the novelty generator is then to use training classes to build a model that can generate objects from future (hold-out) classes, unknown at training time and thus, are novel with respect to the knowledge the model incorporates. Through extensive experiments on various types of generative models, we are able to find architectures and hyperparameter combinations which lead to out-of-distribution novelty.
1 INTRODUCTION
Recent advances in machine learning have renewed interest in artificial creativity. Studies such as deep dream (Mordvintsev et al., 2015) and style transfer (Gatys et al., 2015) have aroused both general public interest and have given strong impetus to use deep learning models in computational creativity research (ICC, 2016). Although creativity has been a topic of interest on and off throughout the years in machine learning (Schmidhuber, 2009), it has been slowly becoming a legitimate sub-domain with the appearance of dedicated research groups such as Google’s Magenta and research work on the topic (Nguyen et al., 2015; Lake et al., 2015).
There is a large body of work studying creativity by computational methods. A large variety of techniques, from rule-based systems to evolutionary computation has been used for a myriad of research questions. Compared to these methods, machine learning methods provide an important advantage: they enable the study of creativity in relation with knowledge (i.e., knowledge-driven creativity; Kazakçı et al. (2016)). Nevertheless, to better highlight the points of interest in computational creativity research for the machine learning community and to allow machine learning researchers to provide systematic and rigorous answers to computational creativity problems, it is important to precisely answer three questions:
1. What is meant by the generation of novelty?
2. How can novelty be generated?
3. How can a model generating novelty be evaluated?
Within the scope of machine learning, it would be tempting to seek answers to these questions in the sub-field on generative modeling. Mainstream generative modeling assumes that there is a phenomena generating the observed data and strive to build a model of that phenomena, which would, for instance, allow generating further observations. Traditional generative modeling considers only in-distribution generation where the goal is to generate objects from the category or categories of
already observed objects. In terms of novelty generation, this can be considered as generating looka-likes of known types of objects. Although there is considerable value in in-distribution generation (e.g., for super-resolution (Freeman et al., 2002; Dong et al., 2014; Ledig et al., 2016) or in-painting (Xie et al., 2012; Cho, 2013; Yeh et al., 2016)), this perspective is limited from a strict point of view of creativity: it is unlikely to come up with a flying ship by generating samples from a distribution of ships and flying objects.
Researchers in creativity research (Runco & Jaeger, 2012) have argued that the crux of creative process is the ability to build new categories based on already known categories. However, creativity is beyond a simple combination exploration: it is about generating previously unknown but meaningful (or valuable) new types of objects using previously acquired knowledge (Hatchuel & Weil, 2009; Kazakçı, 2014). Under this perspective, novelty generation aims at exhibiting an example from a new type. This objective, which we shall call out-of-distribution generation, is beyond what can be formalized within the framework of traditional learning theory, even though learning existing types is a crucial part of the process.
From a machine learning point of view, generating an object from an unknown type is not a welldefined problem, and research in generative modeling usually aims at eliminating this possibility altogether, as this is seen as a source of instability (Goodfellow et al., 2014; Salimans et al., 2016) leading to spurious samples (Bengio et al., 2013). In a way, sampling procedures are designed to kill any possibility of sampling out of the distribution, which is a problem for studying the generation of novelty by machine learning methods.
Arguably, the most important problem is the evaluation of what constitutes a good model for generating out-of-distribution. On the one hand, we are seeking to generate meaningful novelty, not trivial noise. On the other hand, we aim at generating unknown objects, so traditional metrics based on the concept of likelihood are of no use since novelty in the out-of-distribution sense is unlikely by definition. This lack of metrics hinders answering the first two questions. Without a clear-cut evaluation process, the utility of extending the definition of novelty generation to out-of-sample seems pointless.
This paper argues that for a wider adoption of novelty generation as a topic for scientific study within machine learning, a new engineering principle is needed, which would enable such evaluation, and consequently, rigorous experimental research. In the traditional supervised context, the main engineering design principle is the minimization of the error on a hold-out test set. The paper proposes a simple setup where the generative potential of models can be evaluated by holding out entire classes, simulating thus unknown but meaningful novelty. The goal of the novelty generator is then to use training classes to build a model that can generate objects from future (hold-out) classes, unknown at training time.
The main contributions of this paper:
• We design an experimental framework based on hold-out classes to develop and to analyze out-of-distribution generators.
• We review and analyze the most common evaluation techniques from the point of view of measuring out-of-distribution novelty. We argue that likelihood-based techniques inherently limit exploration and novelty generation. We carefully select a couple of measures and demonstrate their applicability for out-of-distribution novelty detection in experiments.
• We run a large-scale experimentation to study the ability of novelty generation of a wide set of different autoencoders and GANs. The goal here is to re-evaluate existing architectures under this new goal in order to open up exploration. Since out-of-distribution novelty generation is arguably a wider (and softer) objective than likelihood-driven sampling from a fixed distribution, existing generative algorithms, designed for this latter goal, constitute a small subset of the algorithms able to generate novelty. The goal is to motivate the reopening some of the closed design questions.
The paper is organized as follows. We review some of the seminal work at the intersection of machine learning and out-of-distribution generation in Section 2. We discuss the conceptual framework of out-of-distribution generation and its relationship with likelihood-based generative models in Section 3. We outline the families of evaluation metrics, focusing on those we use in the paper in Section 4. In Section 4.3 we describe the gist of our experimental setup needed to understand the
metrics described in Section 4.4, designed specifically for the out-of-distribution setup. We describe the details of the experimental setup and analyze our results in Section 5. Finally, we conclude in Section 6.
The paper can be read either in order of the sections, first the motivation and conceptual underpinning of the framework, then the technical contribution, or the other way around, by jumping the Section 4, then coming back to Sections 2 and 3.
2 MACHINE LEARNING AND NOVELTY GENERATION: THE INNOVATION ENGINE, “ZERO-SHOT” LEARNING, AND DISCOVERING NEW TYPES
There are three important papers that consider novelty generation in a machine learning context. Nguyen et al. (2015) propose an innovation engine (Figure 1(a)). They generate images using a neural net that composes synthetic features. The generator is fed back with an entropy-based score (similar to objectness; Section 4.2) coming from an Imagenet classifier, and the feedback is used in an evolutionary optimization loop to drive the generation. An important contribution of the paper is to demonstrate the importance of the objectness score. They show that interesting objects are not generated when asking the machine to generate from a single given class. The generation paths often go through objects from different classes, “stepping stones” which are seemingly unrelated to the final object. The main conceptual difference between our approaches is that Nguyen et al. (2015) do not ground their generative model in learned knowledge: their generation process is not learned model, rather a stochastic combinatorial engine. On the one hand, this makes the generation (evolutionary optimization) rather slow, and on the other, the resulting objects reflect the style of the (preset) synthetic features rather than features extracted from existing objects.
The main goal of Lake et al. (2015) and Rezende et al. (2016) is one-shot learning and generation: learn to classify objects given a small number (often one) of examples coming from a given category, and learn to generate new objects given a single example (Figure 1(b)). One-shot generation is definitely an intermediate step towards out-of-distribution generation. The extremely low number of examples conceptually limits likelihood-based learning/fitting/generation. Lake et al. (2015) circumvents this problem by learning strong Bayesian top-down models (programs) that capture the structural properties of known objects which are generalizable across classes. They also consider unconstrained (“zero-shot”) generation as an extension of their approach, and show that the model can generate new symbols from scratch. They make no attempt to conceptualize the goal of unconstrained generation outside the top-down Bayesian framework, or to design evaluation metrics to assess the quality of these objects, but their intriguing results are one of the strongest motivations of our paper.
Kazakçı et al. (2016) show that symbols of new types can be generated by carefully tuned autoencoders, learned entirely bottom-up, without imposing a top-down Bayesian architecture (Figure 1(c)). They also make a first step of defining the conceptual framework of novelty generation by arguing the goal of generating objects from new types, unknown at the time of training. They design a technique for finding these new types semi-automatically (combining clustering and human labeling). They argue the importance of defining the value of these new types (and of out-of-distribution generation in general), but they make no attempt to design evaluation metrics, thus limiting the exploration and the development of out-of-distribution generative architectures.
3 PROBABILISTIC VS. CONSTRUCTIVE GENERATIVE MODELS
The generative process is commonly framed in a probabilistic setup: it is assumed that an underlying unknown likelihood model P(·) should first be learned on an i.i.d. training sample D = {x1, . . . ,xn}, assumed to be generated from P(·), and then a sampler S should sample from the learned P̂(·). The first step, estimating P(·) using D, is a classical function learning problem that can be studied through the usual concepts of overfitting and regularization, and algorithms can be designed using the classical train/test principle. The second step, designing S for sampling from P̂(·) is also a classical domain of random sampling with a conceptual framework and a plethora of methods.
Technically both steps are notoriously hard for the high-dimensional distributions and the complex dependencies we encounter in interesting domains. Hence, most of the recent and successful methods get rid of the two-step procedure at the level of algorithmic design, and short-cut the procedure from the probabilistic D → P → S to the constructive D → A, where A(D) is a generator, tasked to produce sample objects similar to elements of D but not identical to them. A is fundamentally different from (P,S) in that there is no explicit fitting of a function, we use D to directly design an algorithm or a program.
When the probabilistic setup is still kept for analysis, we face a fundamental problem: if we assume that we are given the true likelihood function P(·), the likelihood of the training sample 1 n ∑n i=1 logP(xi) is a random variable drawn independently from the distribution of log-likelihoods of i.i.d. samples of size n, so the trivial generatorA which resamplesD will have the same expected log-likelihood as an optimal i.i.d. sampler. The resampling “bug” is often referred to as “overfitting”. While it makes perfect sense to talk about overfitting in theD → P → S paradigm (when P is fitted on D), it is somewhat conceptually misleading when there is no fitting step, we propose to call it “memorizing”. When a generator A is trained on D without going through the fitting step D → P , the classical tools for avoiding memorizing (regularization, the train/test framework) may be either conceptually inadequate or they may not lead to an executable engineering design principle.
The conceptual problem of analyzing constructive algorithms in the probabilistic paradigm is not unrelated to our argument of Section 1 that the probabilistic generative framework is too restrictive for studying novelty generation and for designing out-of-distribution generative models. In our view, this flaw is not a minor nuisance which can be fixed by augmenting the likelihood to avoid resampling, rather an inherent property which cannot (or rather, should not) be fixed. The probabilistic framework is designed for generating objects from the distribution of known objects, and this is in an axiomatic contradiction with generating out-of-distribution novelty, objects that are unknown at the moment of assembling a training sample. Resampling (generating exact copies) is only the most glaring demonstration of a deeper problem which is also present in a more subtle way when attempting to generate new types of objects.
We are not arguing that the probabilistic generative framework should be banished, it has a very important role in numerous use cases. Our argument is that it is not adequate for modeling out-ofdistribution novelty generation. What follows from this on the algorithmic level is not revolutionary: the design of most successful generative algorithms already moved beyond the probabilistic framework. On the other hand, moving beyond the probabilistic generative framework at a conceptual level is a paradigm change which will require groundwork for laying the foundations, including revisiting ideas from a domain larger than machine learning.
At the algorithmic/computational level the machine learning community has already started to move beyond likelihood. The overfitting problem is often solved by implicitly constrainingA not to resample. Another common solution is to design tractable likelihood surrogates that implicitly penalize memorization. These surrogates then can be used at the training phase (to obtain non-resampling
generators explicitly) and/or in the evaluation phase (to eliminate generators that resample). The ingenious idea of using discriminators in GANs (Goodfellow et al., 2014; Salimans et al., 2016) is a concrete example; although the setup can be analyzed through the lens of probabilistic sampling, one does not have to fall back onto this framework. If we drop the underlying conceptual probabilistic framework, the constructive GAN idea may be extended beyond generating from the set which is indistinguishable from the set of existing objects. In Section 4.4 we will use discriminators to assess the quality of generators whose very goal is to generate novelty: objects that are distinguishable from existing objects. The main challenge is to avoid the trivial novelty generator, producing uninteresting noise. This challenge is structurally similar to avoiding the trivial memorizing/resampling generator in in-distribution sampling. The two main elements that contribute to the solution is i) to ground the generator strongly in the structure of existing knowledge, without overly fixating it on existing classes, and ii) use a discriminator which knows about out-of-class novelty to steer architectures towards novelty generation.
4 EVALUATION OF GENERATIVE MODELS
In this section we outline the families of evaluation metrics, focusing on those we use in the paper. In Section 4.3 we describe the gist of our experimental setup needed to understand the metrics described in Section 4.4, designed specifically for the out-of-distribution setup.
4.1 INDIRECT SUPERVISED METRICS
When generative models are used as part of a pipeline with a supervised goal, the evaluation is based on the evaluation of the full pipeline. Examples include unsupervised pre-training (Hinton et al. (2006); Bengio et al. (2007); the original goal that reinvigorated research in neural nets), semisupervised learning (Kingma et al., 2014; Rasmus et al., 2015; Maaløe et al., 2016; Salimans et al., 2016), in-painting (Xie et al., 2012; Cho, 2013; Yeh et al., 2016), or super-resolution (Freeman et al., 2002; Dong et al., 2014; Ledig et al., 2016). The design goal becomes straightforward, but the setup is restricted to improving the particular pipeline, and there is no guarantee that those objectives can be transferred between tasks. In our case, the objective of the supervised pipeline may actually suppress novelty. In a certain sense, GANs also fall into this category: the design goal of the generator is to fool a high-quality discriminator, so the generator is asked not to generate new objects which can be easily discriminated from known objects. In our experiments, surprisingly, we found that GANs can be still tuned to generate out-of-distribution novelty, probably due to the deficiencies of both the generator and the discriminator. Our goal in this paper can also be understood as designing a pipeline that turns novelty generation into a supervised task: that of generating objects from classes unknown at training time.
4.1.1 PARZEN DENSITY ESTIMATOR
Parzen density estimators are regularly used for estimating the log-likelihood of a model (Breuleux et al., 2009). A kernel density estimator is fit to generated points, and the model is scored by loglikelihood of a hold-out test set under the kernel density. The metrics can be easily fooled (Theis et al., 2015), nevertheless, we adopted it in this paper for measuring both the in-distribution and out-of-distributions quality of our generators.
4.2 OBJECTNESS
Salimans et al. (2016) proposed a new entropy-based metrics to measure the “objectness”1 of the generated set of objects. As GANs, the metrics uses a trained discriminator, but unlike GANs, it is not trained for separating real objects and generated objects, rather to classify real objects into existing categories. The goal of the generator is create objects which belong confidently to a low number (typically one) of classes. To penalize generators fixating onto single objects or categories, they also require that the set of objects has a high entropy (different objects span the space of the categories represented by the discriminator). The metrics is only indirectly related to classical loglikelihood: in a sense we measure how likely the objects are through the “eye” of a discriminator.
1They also call it “inception score” but we found the term objectness better as it is more general than the single model used in their paper.
Formally, objectness is defined as
1
N n∑ i=1 K∑ `=1 pi,` log pi,` p` ,
where K is the number of classes, pi,` = P(`|xi)
is the posterior probability of category ` given the generated object xi, under the discriminator P trained on a set with known labels, and
p` = 1
n n∑ i=1 pi,`,
are the class marginals.
Salimans et al. (2016) proposed this metric as one of the “tricks” to stabilize GANs, but, interestingly, a similar measure was also used in the context if evolutionary novelty generation (Nguyen et al., 2015).
4.3 ASSESSING OUT-OF-DISTRIBUTION NOVELTY BY OUT-OF-CLASS SCORING
As the classical supervised validation setup simulates past (training) and future (test) by randomly partitioning an existing data set, we can simulate existing knowledge and novelty by partitioning existing data sets holding out entire classes. The goal of the novelty generator is then to use training classes to build a model that can generate objects from future (hold-out) classes, unknown at training. In our first experiments we tried to leave out single classes of MNIST, but the label noise “leaked” hold-out classes which made the evaluation tricky. To avoid this, we decided to challenge the generator, trained on MNIST, to generate letters. We pre-trained various discriminators using different setups, only on digits (MNIST), only on letters (Google fonts), or on a mixture of digits and letters, and used these discriminators to evaluate novelty generators in different ways. For example, we measure in-class objectness and in-class Parzen using a discriminator trained on MNIST, and out-of-class objectness and out-of-class Parzen by a discriminator trained on (only) Google fonts.
4.4 OUT-OF-CLASS SCORES
Naturally, letter discriminators see letters everywhere. Since letters are all they know, they classify everything into one of the letter classes, quite confidently (this “blind spot” phenomenon is exploited by Nguyen et al. (2015) for generating “synthetic” novelty), the letter objectness of an in-distribution digit generator can sometimes be high. For example, a lot of 6s were classified as bs. To avoid this “bias”, we also trained a discriminator on the union of digits and letters, allowing it to choose digits when it felt that the generated object looked more like a digit. We designed two metrics using this discriminator: out-of-class count measures the frequency of confidently classified letters in a generated set, and out-of-class max is the mean (over the set) of the probability of the most likely letter. None of these metrics penalize “fixated” generators, outputting the same few letters all the time, so we combine both metrics with the entropy of the letter posterior (conditioned on being a letter).
Formally, let pi,1, . . . , pi,Kin be the in-class posteriors and pi,Kin+1, . . . , pi,Kin+Kout be the out-ofclass posteriors, where Kin = 10 is the number of in-class classes (digits), and Kout = 26 is the number of out-of-class classes (letters). Let
`∗i = argmax ` pi,`
and `∗outi = argmax
Kin<`≤Kin+Kout pi,`
be the most likely category overall and most likely out-of-class category, respectively. Let
p̃` = ∑n i=1 I {` = `∗outi}∑n
i=1 I {`∗outi > Kin}
be the normalized empirical frequency of the out-of-class category `. We measure the diversity of the generated sample by the normalized entropy of the empirical frequencies
diversity = − 1 logKout Kin+Kout∑ `=Kin p̃` log p̃`,
and define
out-of-class count = (1− λ)× 1 n n∑ i=1 I { `∗i > Kin ∧ pi,`∗i > θ } + λ× diversity,
and
out-of-class max = (1− λ)× 1 n n∑ i=1 pi,`∗outi + λ× diversity.
In our experiments we set the confidence level θ = 0.95 and the mixture coefficient λ = 0.5.
4.5 HUMAN REFEREEING AND THE VISUAL TURING TEST
The ultimate test of l’art pour l’art generative models is whether humans like the generated objects. Visual inspection is often used as an evaluation principle in papers (Denton et al., 2015; Radford et al., 2015; Dosovitskiy et al., 2016), and it is sometimes even made part of the objectified pipeline by using crowdsourcing tools (Denton et al., 2015; Lake et al., 2015; Salimans et al., 2016). First, it definitely makes development (e.g., model selection and hyperparameter tuning) slow. Second, the results depend a lot on what questions are asked and how the responders are primed. For testing generative models, the usual GAN-type question to ask is whether the generated objects are generated by a nature (or a human) or a machine (the visual Turing test). Even those that go the furthest in tasking machines to generate novelty (Lake et al., 2015) ask human judges to differentiate between human and machine. In our view, this question is too restrictive when the goal is out-of-distribution novelty generation. Asking whether an object is “new” is arguably too vague, but inventing adjective categories (such as “surprising” or “interesting” (Schmidhuber, 2009)) that can poll our ability to detect novelty should be on the research agenda. Priming is another important issue: the answer of a human annotator can depend on the information given to her. Nevertheless, a human annotation tool with well-designed priming and questions could accelerate research in novelty generation in the same way labeling tools and standard labeled benchmark sets accelerated supervised learning.
We assessed the visual quality of the set of generated objects using an in-house annotation tool. We took each model which appeared in the top ten by any of the quantitative metrics described in the previous section, and hand-labeled them into one of the following three categories: i) letters, ii) digits, and iii) bad sample (noise or not-a-symbol).
Each panel consisted 26 × 15 generated objects, the fifteen most probable symbols of each letter according to the classifier trained on both letters and digits (Figure 2). The goal of this annotation exercise was i) to assess the visual quality of the generated symbols and ii) to assess the quality of the metrics in evaluating novelty.
5 EXPERIMENTS
Our scores cannot be directly optimized because they all measure out-of-class performance, and showing out-of-class objects at training would be “cheating”. All our (about 1000) models were trained for “classical” objectives: reconstruction error in the case of autoencoders, and adversarial error in the case of GANs. The out-of-class scores were used as a weak feedback for model selection and (quasi random) hyperparameter optimization. The goal is not to be statistically flawless, after all we do not have a statistical model. Rather we set our goal to analyze existing generative architectures from the point of view of novelty generation. Most of the generative models come from a large class of architectures, sometimes purposefully designed for not to “misbehave”. When possible, we turned these tricks, designed to avoid generating “spurious” objects, into optional hyperparameters.
5.1 DETAILED EXPERIMENTAL SETUP
We used two families of deep learning based generative models, autoencoders and GANs. The architectures and the optional features are described in the next sections. All hyperparameters were selected randomly using reasonable priors. All the ∼1000 autoencoders were trained on MNIST training data.
5.1.1 AUTOENCODER ARCHITECTURES AND GENERATION PROCEDURE
We used three regularization strategies for autoencoders: sparse autoencoders (Makhzani & Frey, 2013; 2015), denoising autoencoders (Bengio et al., 2013) and contractive autoencoders (Rifai et al., 2011).
Sparse autoencoders can either be fully connected or convolutional. For fully connected sparse autoencoders, we use the k-sparse formulation from Makhzani & Frey (2013), a simple way of obtaining a sparse representation by sorting hidden units and keeping only the top k%, zeroing out the others, and then backpropagating only through non-zero hidden units.
For convolutional sparse architectures, we use the “winner take all” (WTA) formulation from Makhzani & Frey (2015) which obtains spatial sparsity in convolutional feature maps by keeping only the maximum activation of each feature map, zeroing out the others. We optionally combine it with channel sparsity which, for each position in the feature maps, keeps only the maximum activation across the channels and zero out the others.
For contractive autoencoders, we use the fully connected version with a single hidden layer from Rifai et al. (2011).
We also explore mixtures between the different autoencoder variants in the hyperparameter search. For each model we choose to enable or disable independently the denoising training procedure, the contractive criterion (parametrized by the contractive coefficient, see (Rifai et al., 2011)) and the sparsity rate k (only for fully connected architectures). Table 1 shows the hyperparameters and their priors.
The generation procedure we use for autoencoders is based on Bengio et al. (2013), who proposed a probabilistic interpretation of denoising autoencoders and a way to sample from them using a Markov chain. To have a convergent procedure and to obtain fixed points, we chose to use a deterministic generation procedure instead of a Markov chain (Bahdanau & Jaeger, 2014). As in Bahdanau & Jaeger (2014), we found that the procedure converged quickly.
In initial experiments we found that 100 iterations were sufficient for the majority of models to have convergence so we chose to fix the maximum number of iterations to 100. We also chose to extend
the procedure of Bahdanau & Jaeger (2014) by binarizing (using a threshold) the images after each reconstruction step, as we found that it improved the speed of the convergence and could lead to final samples with an exact zero reconstruction error.
For stochastic gradient optimization of the autoencoder models, we used adadelta (Zeiler, 2012) with a learning rate of 0.1 and a batch size of 128. We used rectified linear units as an activation function for hidden layers in all models. We use the sigmoid activation function for output layers.
5.1.2 GENERATIVE ADVERSARIAL NETWORKS (GANS)
For GANs, we built upon Radford et al. (2015) and used their architecture as a basis for hyperparameter search. We modified the code proposed here to sample new combinations of hyperparameters. Table 2 shows the hyperparameters and their priors.
5.2 ANALYSIS
First, we found that tuning (selecting) generative models for in-distribution generation will make them “memorize” the classes they are trained to sample from. This is of course not surprising, but it is important to note because it means that out-of-class generation is non-trivial, and the vast majority of architectures designed and tuned in the literature are not generating out-of-class novelty naturally. Second, we did succeed to find architectures and hyperparameter combinations which lead to outof-class novelty. Most of the generated objects, of course, were neither digits nor letters (Figure 3), which is why we needed the “supervising” discriminators to find letter-like objects among them. The point is not that all new symbols are letters, that would arguably be an impossible task, but to demonstrate that by opening up the range of generated objects, we do not generate noise, rather objects that can be forming new categories.
The quantitative goal of this study was to assess the quality of the defined metrics in evaluating outof-distribution generators. We proceeded in the following way. We selected the top ten autoencoders and GANs according to the five metrics of out-of-class (letters) count, out-of-class max, out-ofclass objectness, out-of-class Parzen, and in-class Parzen. We then annotated these models into one of the three categories of “letter” (out), “digit” (in), and “bad” (noise or not-a-symbol). The
last three columns of Table 3 show that the out-of-class count and out-of-class max scores work well in selecting good out-of-class generators, especially with respect to in-class generators. They are relatively bad in selecting good generators overall. Symmetrically, out-of-class objectness and the Parzen measures select, with high accuracy, good quality models, but they mix out-of-class and in-class generators (digits and letters). Parzen scores are especially bad at picking good outof-class generators. Somewhat surprisingly, even out-of-class Parzen is picking digits, probably because in-distribution digit generators generate more regular, less noisy images than out-of-class letter generators. In other words, opening the space towards non-digit like “spurious” symbols come at a price of generating less clean symbols which are farther from letters (in a Parzen sense) than clean digits.
We also computed the inter-score correlations in the following way. We first selected the top 10% models for each score because we were after the correlation of the best-performing models . Then we computed the Spearman rank correlation of the scores (so we did not have to deal with different scales and distributions). The first eight columns of Table 3 show that i) in-class and out-of-class measures are anti-correlated, ii) out-of-class count and max are uncorrelated, and are somewhat anti-correlated with out-of-class objectness.
These results suggest that the best strategy is to use out-of-class objectness for selecting good quality models and out-of-class count and max to select models which generate letters. Figure 4 illustrates the results by pangrams (sentences containing all letters) written using the generated symbols. The models (a)-(d) were selected automatically: these were the four models that appeared in the top ten both according to out-of-class objectness and out-of-class counts. Letters of the last sentence (e) were hand-picked by us from letters generated by several top models. Among the four models, three were fully connected autoencoders with sparsity and one was a GAN. All of the three sparse autoencoders had five hidden layers and used a small noise corruption (less than 0.1). The GAN used the default learning rate of 0.0002 and a large number (2048) of fully connected hidden units for the generator, while the number of fully connected hidden units of the discriminator was significantly smaller (128).
6 DISCUSSION AND PERSPECTIVES
In this paper we have proposed a framework for designing and analysing generative models for novelty generation. The quantitative measures make it possible to systematically study the creative capacity of generative models. We believe that human evaluation will remain an important source of feedback in this domain for the foreseeable future. Nevertheless, quantitative measures, such as our out-of-class objectness and out-of-class count and max, will i) make it possible to semi-automate the search for models that exhibit creativity, and ii) allow us to study, from the point of view of novelty generation, the numerous surrogates used for evaluating generative models (Theis et al., 2015), especially those that explicitly aim at quantifying creativity or interestingness (Schmidhuber, 2009).
The main focus of this paper was setting up the experimental pipeline and to analyze various quality metrics, designed to measure out-of-distribution novelty of samples and generative models. The immediate next goal is to analyze the models in a systematic way, to understand what makes them “memorizing” classes and what makes them opening up to generate valuable out-of-distribution samples.
7 ACKNOWLEDGMENTS
This work was partially supported by the HPC Center of Champagne-Ardenne ROMEO.

First, the bad:
This paper is frustratingly written. The grammar is fine, but:
 - The first four pages are completely theoretical and difficult to follow without any concrete examples. These sections would benefit greatly from a common example woven through the different aspects of the theoretical discussion.
 - The ordering of the exposition is also frustrating. I found myself constantly having to refer ahead to figures and back to details that were important but seemingly presented out of order. Perhaps a reordering of some details could fix this. Recommendation: give the most naturally ordered oral presentation of the work and then order the paper similarly.
Finally, the description of the experiments is cursory, and I found myself wondering whether the details omitted were important or not. Including experimental details in a supplementary section could help assuage these fears.
The good:
What the paper does well is to gather together past work on novelty generation and propose a unified framework in which to evaluate past and future models. This is done by repurposing existing generative model evaluation metrics for the task of evaluating novelty. The experiments are basic, but even the basic experiments go beyond previous work in this area (to this reviewer’s knowledge).
Overall I recommend the paper be accepted, but I strongly recommend rewriting some components to make it more digestible. As with other novelty papers, it would be read thoroughly by the interested few, but it is likely to fight an uphill battle against the majority of readers outside the sub-sub-field of novelty generation; for this reason the theory should be made even more intuitive and clear and the experiments and results even more accessible.


FOR THE TASK OF PRODUCT RECOMMENDATION
We propose a unified product embedded representation that is optimized for the task of retrieval-based product recommendation. We generate this representation using Content2Vec, a new deep architecture that merges product content information such as text and image, and we analyze its performance on hard recommendation setups such as cold-start and cross-category recommendations. In the case of a normal recommendation regime where collaborative information signal is available, we merge the product co-occurrence information and propose a second architecture Content2vec+ and show its lift in performance versus non-hybrid approaches in both cold start and normal recommendation regimes.
1 INTRODUCTION
Online product recommendation is now a key driver of demand, not only in E-commerce businesses that recommend physical products, such as Amazon (Marshall, 2006), TaoBao (Xiang, 2013) and Ebay (Academy, 2013), but also in online websites that recommend digital content such as news (Yahoo! - Agarwal et al. (2013), Google - Liu et al. (2010)), movies (Netflix - Bell & Koren (2007)), music (Spotify - Johnson (2015)), videos (YouTube - Covington et al. (2016)) and games (Xbox - Koenigstein et al. (2012)).
Two of the most challenging aspects of recommendation in general and of product recommendation in particular, are scalability and freshness. The first one addresses the problem of making fast recommendations in parallel, the second addresses the problem of updating recommendations based on real-time user interaction. One of the most encountered architecture solutions for recommendation at scale divides the recommendation process in two stages: a candidate generation stage that prunes the number of recommendable items from billions to a couple of hundreds, followed by a second item selection stage that decides the final set of items to be displayed to the user, as shown in Figure 1 (see Mazare (2016), Cheng et al. (2016), Covington et al. (2016)).
The first stage generally implies the pre-generation of an inverted index over the set of recommendable products, paired with a real-time retrieval module, similarly to a search engine architecture. In our current paper we focus on the cases where the system supports vectorial product queries. The sources of the vectorial representations range from the set of co-occurring products, like in the case of neighborhood-based collaborative filtering, to a low-dimensional representation produced via matrix factorization or to an embedded representation produced via a deep neural network.
The second stage takes the candidate set and decides the final list of recommendations, usually by optimizing a ranking metric. This stage has in general a lot more constraints in terms of latency, due to its use of real-time signal that makes its predictions not cacheable. Therefore, in terms of model choice, the first stage can be a lot more complex than the second. In terms of impact, the quality of the candidate set coming from the first stage is crucial, since this constitutes a hard threshold on the performance of the second stage and of the overall system.
Because of the feasibility of using a more complex model and the potential impact on the final recommendation performance, we choose to concentrate our efforts on the task of optimal candi-
date generation. We formalize the problem as a link prediction task, where given a set of past co-purchased products we try to predict unseen pairs of products. Related work in representation learning for recommendation investigated the use of collaborative filtering (CF), text and product images, but to our knowledge, there has been no attempt to unify all of these signals in a single representation. We see this as an opportunity to investigate the leveraging effect of generating a Unified Product Representation via a deep-learning approach. In the following, we formally define the set of associated requirements we would like to satisfy:
• Relevance: the representation should be optimized for product recommendation relevance, as measured by the associated target metrics (in this case, modeling it as a link prediction task and optimizing for the AUC of product pair prediction).
• Coverage: the representation should leverage all available product information (in our case, all product information available in the product catalog together with observed product co-occurrences).
• Cross-modality expressiveness: the representation should be able to account for interactions between various information sources such as text and image (can take into account the fact that the word ”red” and the ”red” color detector are correlated).
• Pair-wise expressiveness: the representation should be able to account for interactions between the two products.
• Robustness: the representation should operate well (recommendation performance will not degrade dramatically) in hard recommendation situations such as product cold-start (new products, new product pairs) and cross-category recommendation. These are important use-cases in product recommendation, when the product catalog has high churn (as in the case of flash sales websites or classifieds) or the recommendation needs to leverage crossadvertiser signal (as in the case of new users and user acquisition advertising campaigns). This is a different goal from simply trying to optimize for relevance metrics, due to the inherent limitations of offline metrics in predicting future online performance.
• Retrieval-optimized: the representation should be adapted to a content-retrieval setup, both on the query and on the indexing side, meaning that the vectors should be either small, sparse or both.
We propose a modular deep architecture that leverages state-of-the-art architectures for generating embedded representations for image, text and CF input, re-specializes the resulting product embeddings and combines them into a single product vector. This is a very general architecture that can plugin any networks in the image and text domain and re-use them for the problem of product recommendation, along with their gains in representation learning for the two domains. We investigate multiple ways of merging the modality-specific product information and propose a new type of residual-inspired unit, which we name Pairwise Residual Unit, that can model the joint aspects of the different product embeddings and show that it leads to good improvements.
We analyze our proposed architecture on an Amazon dataset (McAuley et al., 2015) containing information on co-purchased products. We report our improvements versus a text and an imagebased baseline, that was introduced in previous work by (cite Julian) and show improvements both on normal and hard recommendation regimes such as cold-start and cross-category setups.
Our approach is similar to the recent work by (Covington et al., 2016), that propose a solution for video recommendation at YouTube. Unlike their proposed solution, where, in order to support user vector queries, the candidate generation step co-embeds users and items, we are interested to coembed just the product pairs, which generally has a much smaller dimension. In our approach, the personalization step can happen after the per-item candidates are retrieved.
Our main contributions are the following:
• We propose a novel way of integrating deep-learning item representation in the context of large scale recommender system with a 2-stage serving architecture and introduce the new task of Unified Product Representation for optimal candidate selection in both cold start and normal recommendation setups.
• We introduce a new deep architecture that merges content and CF signal for the task of product recommendation and propose the Pairwise Residual Unit, a new learning component that models the joint product representations.
• We introduce two novel experimental setups (hard cold start, cross-category) and test that the proposed Content2Vec architecture satisfies the requirements we defined.
Though the focus of our work is on improving product recommendation through representation learning, we believe that simple extensions of our approach can be applied to many other recommendation scenarios.
The rest of the paper goes as follows: In Section 2 we cover previous related work and the relationship with our method. In Section 3 we present the Content2Vec model, followed by a detailed description of the resulting architecture in Section 4. In Section 5 we present the experimental setup and go over the results on Section 5.2. In Section 6 we summarize our findings and conclude with future directions of research.
2 RELATED WORK
Our work fits in the new wave of deep learning based recommendation solutions, that similarly to classical approaches can fall into 3 categories, namely collaborative filtering based, content based or hybrid approaches.
Several approaches use neural networks to build better item representations based on the cooccurrence matrix. The Prod2Vec algorithm (see (Grbovic et al., 2015)) implements Word2Vec ((Mikolov et al., 2013a), (Shazeer et al., 2016)), an algorithm that is at origin a shallow neural language model, on sequences of product ids, to reach a low-dimensional representation of each product. Among other embedding solutions that use the item relationship graph are the more recent extensions to Word2Vec algorithm such as Glove (Pennington et al., 2014) and SWIVEL (Shazeer et al., 2016) and the graph embedding solutions proposed in Node2Vec (Grover & Leskovec, 2016) and SDNE (Wang et al., 2016).
Content-based methods recommend an item to a user based upon an item description and a user profile ((Pazzani & Billsus, 2007)). This idea was deeply investigated in the information retrieval literature: in the context of web search, DSSM (Huang et al., 2013) and its extensions (Shen et al., 2014)(C-DSSM) and (Shan et al., 2016) are some of the most successful methods that specialize
query and document text embedding in order to predict implicit feedback signal such as document click-through rate. In the context of product recommendation, in (McAuley et al., 2015) the authors feed a pre-trained CNN (CNN trained on the ImageNet dataset, which is an image classification task that is very different from the task of image-based product recommendation) with products images and use the last layer of the network as the product embedding. This representation is subsequently used to compute similarities between products. Similarly, the authors in (Van den Oord et al., 2013) use CNNs to compute similarities between songs. Yosinski et al. (2014) show that the low layers of DNNs trained on different tasks are often similar and that good performance can be reached by fine-tuning a network previously trained on another task. In the case of recommendation systems, this fine tuning was implemented in Veit et al. (2015), where the authors specialize a GoogLeNet architecture to the task of predicting co-view events based on product pictures.
The performance of Collaborative Filtering (CF) models is often higher than that of content-based ones but it suffers from the cold-start problem. To take advantage of the best of both worlds, hybrid models use both sources of information in order to make recommendations. One possible way to incorporate product information is using it as side information in the product sequence model, as proposed in Meta-Prod2Vec (Vasile et al., 2016), leading to better product embeddings for products with low signal (low number of co-occurrences). In this work we continue the investigation of using both types of signal, this time both at training and product recommendation time.
3 CONTENT2VEC MODEL
Our proposed approach takes the idea of specializing the input representations to the recommendation task and generalizes it for multi-modality inputs, in order to leverage all product information and in particular, product images and product title and description text.
The main criteria for the Content2Vec architecture is to allow us to easily plugin new sources of signal and to replace existing embedding solutions with new versions. We are also interested in separating product-level embeddings from pair-level embeddings, such that the network can generate product vectors that are readily indexable. As a result, the Content2Vec architecture has three types of modules, as shown in Figure 2:
• Content-specific embedding modules that take raw product information and generate the associated vectors. In this paper we cover embedding modules for text, image, categorical attributes and product co-occurrences (for an example, see Figure 3).
• Overall product embedding modules that merge all the product information into a unified product representation.
• Pair embedding module that merges the product-to-product interactions and computes the final similarity score. In the case of retrieval-optimized product embeddings, this module becomes the inner-product between the two items and all interactions between them are to be approximated within the product-level embedding modules.
Content2Vec training follows the architecture, learning module-by-module. In the first stage, we initialize the content-specific modules with embeddings from proxy tasks (classification for image, language modeling for text) and re-specialize them to the task of product recommendation. For the specialization task, as mentioned in Section 1, we frame the objective as a link prediction task where we try to predict the pairs of products purchased together. We describe the loss function in Section 3.1. In the second stage, we stack the modality-specific embeddings generated in the first stage into a general product vector and learn an additional residual vector using the same learning objective as in the specialization step. This will described in depth in Section 4.2. Finally, in the third stage, given the updated product vectors from stage two, we learn the linear combination between the similarities of the product vectors and make the final prediction.
3.1 LOSS FUNCTION
The previous work on learning pair-wise item distances concentrated on using ranking (McFee & Lanckriet, 2010), siamese (Hadsell et al., 2006) or logistic loss (Zheng et al., 2015). For optimizing the link prediction objective we choose the logistic similarity loss (eq. 1) that has the advantage of
having a fast approximation via Negative Sampling loss (Mikolov et al., 2013b) shown in eq. 2. By using Negative Sampling, the prediction step can scale up to large number of items, by using all positive pairs and sampling the negatives on the fly.
L(θ) = ∑ ij −XPOSij log σ(sim(ai, bj))−XNEGij log σ(−sim(ai, bj)) (1)
LNS(θ) = ∑ ij −XPOSij (log σ(sim(ai, bj)) + k∑ l=1 Enl∼PD log σ(−sim(ai, nl))) (2)
where: θ = (ai, bj) is the set of model parameters, where ai and bj are the embedding vectors for the products A and B, sim(ai, bj) = α < ai, bj > +β is the similarity function between ai and bj , α and β are scalar values, XPOSij is the frequency of the observed item pair ij (e.g. the frequency of the positive pair ij), XNEGij = Xi − XPOSij is the frequency of the unobserved item pair ij (we assume that all unobserved pairs are negatives), PD probability distribution used to sample negative context examples nl, k is a hyper parameter specifying the number of negative examples per positive example.
4 CONTENT2VEC MODULES
4.1 CONTENT-SPECIFIC EMBEDDING MODULES
Content-specific modules can have various architectures and are meant to be used separately in order to increase modularity. Their role is to map all types of item signal into embedded representations.
In Figure 3 we give an illustrative example of mapping a pair of products to their vectorial representations.
In the following we analyze four types of input signal and embedding solutions for each one of them. For all of the modules, we use LNS loss (see eq. 2) as specialization loss.
4.1.1 EMBEDDING PRODUCT IMAGES: ALEXNET
Model and proxy task: CNN for Image Classification For generating the image embeddings we propose reusing a model trained for image classification, as in previous work by (Krizhevsky et al., 2012) and (He & McAuley, 2015). In (He & McAuley, 2015), the authors have shown how to use the Inception architecture (Szegedy et al., 2015) and specialize it for the product recommendation task. However, the Inception architecture is very deep and requires extensive training time. For ease of experimentation we use AlexNet, which is a simpler architecture that was also a winner on the ImageNet task (Krizhevsky et al., 2012) previously to Inception NN. In section 5.2 we will show that, even if simpler, when combined with additional product text information, the AlexNet-based solution can perform very well on the recommendation task. For our experiments, we use the pretrained version of AlexNet available on Toronto’s university website. We experimented with two different ways to specialize the representation in order to compute product similarities. In the first one, we learn a weighted inner product between the two representations (fc7 layer of ImageNet). In the second one, we specialize the fc7 layer to detect product similarities. The second approach lead to much better performance and is the one for which we report final results.
4.1.2 EMBEDDING PRODUCT TEXT: WORD2VEC AND CNN ON SENTENCES
Model and proxy task: Word2Vec for Product Language Modeling For generating word embeddings, we propose reusing Word2Vec Mikolov et al. (2013b), a model for generating language models that has been employed in a various of text understanding tasks. More recently, it has been shown in (Pennington et al., 2014) that Word2Vec is closely linked with matrix factorization techniques applied on the word co-occurrence matrix. For Content2Vec, we chose to pretrain Word2Vec
on the entire product catalog text information and not use an available set of word embeddings such as the one created on the Google Corpus. The main reason is that the text distribution within product descriptions is quite different from the general distribution. For example the word ’jersey’ has a very different conditional distribution within the product description corpus versus general online text. Text CNN (Kim, 2014) offers a simple solution for sentence-level embeddings using convolutions. The convolutions act as a form of n-gram filters, allowing the network to embed sentence-level information and specializing word embeddings to higher-order tasks such as text classification or sentiment analysis. To the best of our knowledge, this is the first attempt to employ them for the task of product recommendation. For our task, we generate sentences based on the product titles and descriptions.
4.1.3 EMBEDDING PRODUCT CO-OCCURRENCES: PROD2VEC
Prod2Vec (Grbovic et al., 2015) is an extension of the Word2Vec algorithm to product shopping sequences. As a result, Prod2Vec can be seen as a matrix factorization technique on the product co-occurence matrix. In Content2Vec, the Prod2Vec-based similarity contains all of the information that can be derived from the sequential aspect of the user behavior, without taking into account the per-product meta-data.
4.1.4 EMBEDDING CATEGORICAL PRODUCT META-DATA: META-PROD2VEC
Meta-Prod2Vec (Vasile et al., 2016) improves upon Prod2Vec by using the product meta-data side information to regularize the final product embeddings. In Content2Vec, we can use the similar technique of co-embedding product categorical information with product ids to generate the embedding values for the categorical features.
4.2 JOINT PRODUCT EMBEDDING: PAIRWISE RESIDUAL UNIT
As stated in Section 1, the function of the product embedding module is two-fold: first, to model all interactions that exist between the modality-specific embeddings with respect to the final optimization objective, and second, to approximate interaction terms between the products that cannot be explained by a linear combination of the modality-specific similarities. With this in mind, we introduce a new type of learning unit, the Pairwise Residual Unit (eq. 4), which similarly to the original residual unit introduced in He et al. (2015) (eq. 3), allows the layers to learn incremental, i.e. residual representations (see Figure 4). In Hardt & Ma (2016) the authors motivate the use of residual units as helping preserve the representations learned in the previous layers. In our case we are interested in preserving the specialized image and text representations and learn an additional representation for their interactions. Though in previous work, most the of the residual units are using at least two ReLU layers in the residual unit, we observe good results using just one. In order to model interactions between modalities, we could also learn a fully connected layer initialized with identity that takes as input the concatenated modality-specific vectors. However, in order to have a smaller number of parameters and increase model comprehensibility, we would like to keep separate the modality-specific representations and to model the final prediction model as an ensemble.
y = F (x) + x (3)
y = sim(F (x1), F (x2)) + sim(x1, x2) (4) where: x1 and x2 are the two product embedding vectors (obtained by stacking the modality-specific vectors), sim(., .) is a similarity function over two embedding vectors x1, x2, F (x) is a Rectified Linear Unit.
To be able to measure the incremental value of introducing a residual vector we introduce a baseline architecture that computes the final prediction based on the linear combination of the modalityspecific similarities denoted by Content2Vec-linear with the associated similarity function defined in eq. 5.
simc2v(ai, bj) = ∑
m∈Modalities
wmσ(simm(ai, bj)) (5)
Under this notation, the residual-based architecture denoted as Content2Vec-res minimizes LNS with the similarity function defined in eq. 6.
simc2v−res(ai, bj) = ∑
m∈(Modalities+Residual)
wmσ(simm(ai, bj)) (6)
In order to learn the residual vector, we keep fixed the modality-specific similarities and co-train the final weights of each of the modalities together with the product-specific residual layers. For example, in the case of using only image and text signals, our final predictor can be defined as in eq 7, where Ptxt and Pimg are pre-set and wtxt, wimg , wres and Pres are learned together:
P (pos|a, b) = σ(wtxtPtxt(pos|atxt, btxt) + wimgPimg(pos|aimg, bimg) + wresPres(pos|ares, bres)) (7)
where: pos is the positive outcome of products A and B being bought together and Pres(pos|a, b) = σ(α < F ([atxt, aimg]), F ([btxt, bimg]) > +β)
In Section 5.2 we compare the performance of Content2Vec-res and Content2Vec-linear and show that, as expected, the proposed architecture surpasses the performance of the linear model, while allowing for a retrieval-based candidate scoring solution.
4.3 PAIR EMBEDDING MODULE
In a retrieval-based architecture, the pair embedding module cannot support more than a simple linear combination of the product embedding vectors, such that the final score can be computed via inner-product. However, we are still interested to know the trade-off in performance between an inner-product-based candidate scoring and a model that allows for explicit interaction terms between the items. To this end, we introduce two explicit interaction models: Content2Vec-crossfeat - a
model where we discretize the text and image-specific similarity scores and create explicit feature conjunctions between them and Content2Vec-embedpairs - a model where we use a similar technique with Paiwise Residual Unit, in this case modeling the residual of the linear similarity directly as a vector in the pair embedding layer, as shown in Figure 5. In Section 5.2 we show that two models have as expected better performance than the linear model and that the pair embedding is slightly better.
5 EXPERIMENTAL RESULTS
5.1 DATASET
We perform our evaluation on the publicly available Amazon dataset (McAuley et al., 2015) that represents a collection of products that were co-bought on the Amazon website. Each item has a rich description containing product image, text and category (any of the modalities can be missing). In terms of dimensionality, the dataset contains around 10M pairs of products. We concentrate on the subgraph of Book and Movie product pairs, because both categories are large and they have a reasonable sized intersection. This allows us to look at recommendation performance on crosscategory pairs (to evaluate a model trained only on Book pairs on predicting Movie co-bought items) and mixed category pairs (to evaluate the models on Book-Movie product pairs).
Based on the full Book & Movies data we generate three datasets with different characteristics: The first dataset simulates a hard cold start regime, where all product pairs used in validation and testing are over products unseen in training. This tests the hardest recommendation setup, where all testing data is new. We decided to bench all of our hyperparameters on this regime and use the best setup on all datasets, since tuning on the harder dataset ensures the best generalization error (results shown in Table 1). The second dataset simulates a non-cold start regime, where the vast majority of the products in the test set are available at training time. The dataset is generated by taking the top 100k most connected products in the original dataset and keeping the links between them (results shown in Table 2). The third dataset simulates a soft cold start regime, where some of the products in the test set are available at training time. The dataset is generated by taking the top 200k most connected products in the original dataset and sampling 10% of the links between them (results shown in Table 3).
Hyper-parameters We fixed the sizes of embedding vectors for image CNN module to 4096 hidden units, for text CNN module to 256, for Prod2Vec module to 50, for residual representation to 128. For optimization we use an Adam algorithm and we manually set the initial learning rate based on the validation set performance. The batch sizes vary for different datasets. We train all the models until validation set performance stops increasing.
Evaluation task We evaluate the recommendation methods on the product link prediction task, similar to (He & McAuley, 2015). We consider the observed product pairs as positive examples and all unknown pairs as negatives. We generate negative pairs according to the popularity of the products in the positive pairs (negative examples between popular products are more likely to be generated) with a positive to negative ratio of 1:2.
Evaluation metrics For the link prediction task, we use the Area Under Curve (AUC) of the Precision/Recall curve as our evaluation metric.
Competing methods
• ImageCNN: prediction based on specialized image embeddings similarity • TextCNN: prediction based on specialized text embeddings similarity • Content2Vec-linear: prediction based on the linear combination of text and image similar-
ities • Content2Vec-crossfeat: prediction based on the linear combination of discretized image
and text similarities and their conjuctions • Content2Vec-res: prediction based on the linear combination of text and image similarities
plus product-level residual vectors similarities • Content2Vec-embedpairs: prediction based on the linear combination of text and image
similarities and a pair-level residual component • Prod2Vec: prediction based on the product vectors coming from the decomposition of the
co-purchase matrix • Content2Vec+: prediction based on the ensemble of Prod2Vec and Content2Vec models
5.2 RESULTS
The results on hard and soft cold start datasets (Tables 1, 3) show that our main proposed method Content2Vec-res can leverage the additional signal provided by each of the input modalities in a joint manner and leads to significant gains in AUC versus the one-signal baselines (ImageCNN, TextCNN) and their linear combination (Content2Vec-linear). From the point of view of robustness, Content2Vec-res learns product representations that perform better than the baseline methods on out-of-sample recommendations such as cross-category pairs and mixed-category pairs (Table 1). We observe that adding an additional layer that represents pair-level interactions does not lead to big improvements in either of the two models we investigated (Content2Vec-crossfeat,embedpairs), confirming that a product retrieval-based recommender system can achieve state-of-the-art results. Finally, Content2Vec-res+, our proposed hybrid architecture that combines content and CF signal achieves better performance than the content and CF-only models, with bigger lifts in the case of the third dataset (Table 3) where the CF signal is weaker due to higher sparsity.
Recommendation Model Test Content2Vec-linear 84% Content2Vec-res 87% Prod2Vec 96% Content2Vec-linear+ 97% Content2Vec-res+ 97%
Table 2: AUC results on non cold-start dataset.
Recommendation Model Test ImageCNN 80% TextCNN 78% Content2vec-linear 88% Content2vec-res 89% Content2vec-embed pairs 90% Prod2vec 86% Content2vec-linear+ 89% Content2vec-res+ 92% Content2vec-embed pairs+ 92%
Table 3: AUC results on soft cold-start dataset.
6 CONCLUSIONS
This work has several key contributions. We show how to use all product signal for the task of product recommendation using a modular architecture that can leverage fast evolving solutions for each type of input modality. We define a set of requirements for evaluating the resulting product embeddings and show that our method leads to significant improvements over the single signal approaches on hard recommendation situations such as cold-start and cross-category evaluation. Finally, in order to model the joint aspects of the product embeddings we introduce a new type of learning unit, named Pairwise Residual Unit and show the resulting gains on a real product co-purchases dataset. In the current work we have addressed all but one of the desired requirements, namely generating retrieval-optimized embeddings. For the next steps, we want to pursue sparse and compressed product representations, in order to help the performance of the final product retrieval system.

The problem of utilizing all available information (across modalities) about a product to learn a meaningful "joint" embedding is an interesting one, and certainly seems like it a promising direction for improving recommender systems, especially in the "cold start" scenario. I'm unaware of approaches combining as many modalities as proposed in this paper, so an effective solution could indeed be significant. However, there are many aspects of the proposed architecture that seem sub-optimal to me:
1. A major benefit of neural-network based systems is that the entire system can be trained end-to-end, jointly. The proposed approach sticks together largely pre-trained modules for different modalities... this can be justifiable when there is very little training data available on which to train jointly. With 10M product pairs, however, this doesn't seem to be the case for the Amazon dataset (although I haven't worked with this dataset myself so perhaps I'm missing something... either way it's not discussed at all in the paper). I consider the lack of a jointly fine-tuned model a major shortcoming of the proposed approach.
2. The discussion of "pairwise residual units" is confusing and not well-motivated. The residual formulation (if I understand it correctly) applies a ReLU layer to the concatenation of the modality specific embeddings, giving a new similarity (after dot products) that can be added to the similarity obtained from the concatenation directly. Why not just have an additional fully-connected layer that mixes the modality specific embeddings to form a final embedding (perhaps of lower dimensionality)? This should at least be presented as a baseline, if the pairwise residual unit is claimed as a contribution... I don't find the provided explanation convincing (in what way does the residual approach reduce parameter count?).
3. More minor: The choice of TextCNN for the text embedding vectors seems fine (although I wonder how an LSTM-based approach would perform)... However the details surrounding how it is used are obscured in the paper. In response to a question, the authors mention that it runs on the concatenation of the first 10 words of the title and product description. Especially for the description, this seems insufficiently long to contain a lot of information to me.
More care could be given to motivating the choices made in the paper. Finally, I'm not familiar with state of the art on this dataset... do the comparisons accurately reflect it? It seems only one competing technique is presented, with none on the more challenging cold-start scenarios.
Minor detail: In the second paragraph of page 3, there is a reference that just says (cite Julian).