Showing items that do not match search query intent degrades customer experience in e-commerce. These mismatches result from counterfactual biases of the ranking algorithms toward noisy behavioral signals such as clicks and purchases in the search logs. Mitigating the problem requires a large labeled dataset, which is expensive and time-consuming to obtain. In this paper, we develop a deep, end-to-end model that learns to effectively classify mismatches and to generate hard mismatched examples to improve the classifier. We train the model end-to-end by introducing a latent variable into the cross-entropy loss that alternates between using the real and generated samples. This not only makes the classifier more robust but also boosts the overall ranking performance. Our model achieves a relative gain compared to baselines by over 26% in F-score, and over 17% in Area Under PR curve. On live search traffic, our model gains significant improvement in multiple countries. Deep learning models have shown excellent performance in the natural language domain, and this success has inspired practitioners to adapt these models to information retrieval tasks (Mitra et al., 2017; Huang et al., 2013). However, deep learning has not succeeded in these tasks due to the lack of massive labeled datasets (Dehghani et al., 2017). Another reason is that word-based representations (Mikolov et al., 2013; Pennington et al., 2014) are less useful in representing complex, informal search queries (Xiong et al., 2017) and hence provide limited understanding of the search intent. In the absence of explicit knowledge of which documents are matched with a search query and which are mismatched, it is hard to learn robust deep learning models that understand the query intent and find high-quality, relevant documents. Text-based product search is even more challenging. Simple modifications to the input query (or a product title) can completely change the search intent (or the product type, respectively). Take, for example, the query gray iPhone X by which a user is looking for a specific phone. Slightly modified queries such as iPhone X charger and case for iPhone X refer to different products. Therefore, it is hard for distributed representations to capture the nuances. Moreover, noisy user-behavioral signals from clicks and purchases (e.g., users purchased a phone while searching for a charger) can lead to biases in the ranking algorithms. As such, even top-ranked items may not match the search intent. In this paper, we consider the problem of identifying query-item mismatches to enhance the ranking performance in product search. This task typically requires a large labeled dataset of matches and mismatches that we will respectively refer to as negative and positive samples. Even if we can partly afford the expensive and time-consuming labeling, acquired datasets are unbalanced and lack hard positive samples, preventing the classifier from learning a robust decision boundary. However, the above examples gray Iphone X and Iphone X charger motivate that meaningful positive samples can be artificially generated by leveraging the labeled data. In fact, we can heuristically construct a large number of negatives by observing which items are commonly purchased in response to the corresponding query. The question is that can we use such negatives to synthesize hard-to-classify positives to robustify the classifier? We illustrate the goal of the generation in Figure 1. To this end, we develop a deep, end-to-end model that learns to identify mismatched query-item pairs and is also capable of generating mismatched queries given an item. The task of the generator is twofold: it has to be able to generate hard-to-classify samples so that the classifier Figure 1: (Best seen in color) The query is running shoes for men . learns a more robust decision boundary; it also needs to generate realistic queries. Using matched query-item pairs allows the generator to synthesize hard-to-classify mismatches based on an efficient encoder-decoder architecture. This has a distinct advantage over generating samples from noise, as in Generative Adversarial Networks (Goodfellow et al., 2014; Wang et al., 2017) or via dithering the learned representations to make the model more robust (Miyato et al., 2018). We include our classifier and generator in an end-to-end model. The classifier only requires continuous representations of the generated query as the second input instead of a discrete text sequence. This key property enables us to use efficient gradient-based optimization techniques and bypass reinforcement learning-based methods (Jia and Liang, 2017), which are significantly more complex, and also recently developed heuristic approaches to generate adversarial text samples (Alzantot et al., 2018). To achieve this, we modify the objective function in a way that makes the end-to-end training possible via sampling a binary latent variable, avoiding the min-max optimization for GANs (Miyato et al., 2018; Wang et al., 2017). We perform extensive experiments on a mismatch dataset in an e-commerce company. The proposed model outperforms deep learning baselines by over 26% in F-score and 17% in relative AUPR score and performs significantly better than GBDT models, which are widely used in practice. Including the query generator helps achieve higher gains than merely dithering the vector representation of the query. We also show that the generative model can indeed generate hard-to-classify mismatches. When integrated with the ranking component of a real-world product search engine, our model outperforms the baseline methods in multiple countries on an online A/B test evaluation. Let x = ( I, Q ) denote a pair of item title and textual query and y ( I, Q ) denote its corresponding label. y = 1 if the pair is mismatched or y = 0 otherwise. Assume we can obtain from search logs many matched samples, which we use to generate more positives. These samples are not human-labeled but instead inferred by considering behavioral signals such as frequent purchases. We aim to build a deep classifier that takes two text sequences in x i = ( I i , Q i ) and classifies whether the pair is mismatched or not. At the same time, we want the model to generate a new sample ( I, Q gen ) with y gen = 1 given ( I, Q ) with y = 0 . Next, we discuss our proposed model. We present our proposed model, namely QUARTS (QUery-based Adversarial learning for Robust Textual Search) in Figure 2. QUARTS is composed of three components: (i) an LSTM and attention-based classifier, (ii) a variational encoder-decoder query generator (VED) and (iii) a state combiner. Due to space constraints, we defer the details of (i) and (ii) in the appendix. The LSTM classifier (i) is adapted from the entailment model in (Rocktaschel et al., 2015), with some changes to fit the product search task (see Appendix A.1). The VED generator (ii) takes a matched pair ( I, Q ) as input and outputs a new query Q gen so that the pair ( I, Q gen ) is mismatched while Q gen stays lexically similar to Q . As an example, if I = Apple Iphone X, space gray and Q = gray Iphone X is a matched pair, we can generate Q gen = Iphone X case given I . In this case, Q gen is similar to Q , but ( I, Q gen ) constitutes a product mismatch. To have an end-to-end model, we combine the query representations computed by the classifier and the generator to form a proper input to the attention layer. We need to make sure that the modifications still allow us to efficiently backpropagate the gradients of the loss function during training. To achieve this, we add a merging layer shown by the orange box in Figure 2. This layer computes sH gen + (1  s ) H, s = (1  y ( I, Q )) z where H, H gen are the corresponding LSTM representations of the input Q and Q gen , and z  Bernoulli ( p ) is a random binary variable that controls whether the input query Q or the generated query Q gen is used. When z = 0 , QUARTS essentially computes the probability of mismatch. Let us explain how the real label y and the switch z combine to yield the desired outputs. As y = 1 where the sample ( I, Q ) is a real positive, we want to leverage it to train the classifier f  (  ) . In this case, s = 0 and the attention layer only takes H as input. When y = 0 , we can either use this sample to train the classifier or use it to generate adversarial representations H gen . This process is controlled by z . When z = 1 , we use H gen , else H . The value of z determines whether we want to use the datapoint as-is for training, or instead use the fake query via the VED module. A second consideration is how to enable efficient training on f  (  ) and the generator g  (  ) . Let x gen = ( I, Q gen ) be the datapoint we will use to train f  (  ) using the output from g  (  ) . In this case, since y = 0 , z = 1 , we use z as a proxy label to train f  (  ) . For samples i = 1 , 2 , . . . , N , we sample z i  Bernoulli ( p ) for some p  [0 , 1) to decide which negative samples have labels flipped. We modify the cross entropy loss as below, with L  being the weighted cross-entropy loss: 1 NN (cid:88) i =1 (1  s i ) L  ( x i , y i )+ s i L  ( g  ( x i ) , z i )) . (1) Note that (1) is differentiable in ,  and notably H gen  the generated representations of Q gen . Since we do not use the actual generated query, we need not resort to heuristics or policy gradient-based optimization methods to minimize (1). Before training QUARTS end-to-end, we pre-train the classifier and the VED on proper data. The pseudocode of the end-to-end training is shown in Algorithm 1. Algorithm 1 QUARTS training procedure Require: N samples of labeled data ( I, Q, y ( I, Q )) , M negative samples from search log, and sampling probability p 1: Using labeled data, pre-train the classifier 2: Create ( I, Q, Q mis ) tuples T using labeled data so that y ( I, Q ) = 0 and y ( I, Q mis ) = 1 3: Initialize the VED encoder with the trained classifier, and use the above created tuples to pre-train the VED generator 4: Concatenate the human annotated and logs data to form M + N samples D 5: Perform end to end training on D , where in each epoch 6: for i  [ M + N ] do 7: Sample z  Bernoulli ( p ) 8: Set s = (1  y i ( I, Q )) z 9: Use s and I, Q, y ( I, Q ) to perform one step of learning on the end-to-end model 10: end for 3 Experiments and Results We used a human-labeled dataset of query-item pairs, obtained from an e-commerce search platform. There are in total N = 3 . 2 M pairs of which only a small fraction are mismatches. A separate test set of  100 K labeled pairs was used to evaluate all methods. We further have 3 M query-item pairs that are deemed matched by considering items that are purchased frequently in response to those queries from the search logs. This acts as the augmentation dataset for the QUARTS model. For all encoders and decoders, we use an LSTM with hidden size of 300. The inputs to the encoder are 300 dimensional word embeddings trained separately for queries and item titles. The word embeddings were trained using word2vec on a corpus of anonymized search engine queries, as well as item titles from the catalog. The models were trained using Adam (Kingma and Ba, 2014) and we tuned the classification part (i.e. excluding the variational decoder) on a validation dataset. We obtained the performance with initial learning rate 10  4 , and learning rate decay 0 . 8 after 10 epochs. The dropout probability and the batch size were respectively 0 . 1 and 128 . Because the imbalanced nature of the labeled data, we up-weighted the positive samples. In the cross-entropy loss for classification, we set  = 5 . To pretrain the VED, we used the annotated training data and generated I, Q, Q gen tuples as explained in Section A.2. Since we are explicitly interested in training the VED to generate Q gen : y ( I, Q gen ) = 1 given I, Q : y ( I, Q ) = 0 , we consider only the annotated items that have both positive and negatively annotated queries, and generate the tuples. The previously pretrained encoder was fixed, and only the decoder was trained using Adam with an initial learning rate of 10  3 . We finally merged the LSTM encoder for query and item, the VED decoder for query with the other layers described in the previous sections to train the model end to end. The classifier f  (  ) is pretrained on the human annotated data. For the end-to-end model, we use the pretrained classifier and generator, modify the loss function as in (1), and further append the dataset with M = 3 MM well matched items from anonymized user logs, where we assume items that were purchased in response to a query are matched ( y ( I, Q ) = 0) . We evaluated our models using Area under the Precision-Recall curve (APR), and the F1-score at the best operating point, all evaluated on the test set. To evaluate the generation task, we used BLEU scores. In addition, we had human annotators to judge generated item-query pairs. These annotators were trained to identify whether a generated pair is a match or a mismatch. We used a GBDT model as a baseline. We used user-item features for this model similarly to traditional ranking and relevance models. We also applied a DSSM-style model (namely DSSM) where query and item word embeddings were concatenated as input to a stack of dense layers. We also used the BERT (Devlin et al., 2018) embeddings for the query and item title sequences and passed them through the aforementioned model. A final baseline we evaluated against was the MatchPyramid (Pang et al., 2016), which has shown to outperform several baselines for matching and question-answering tasks. All hyperparameters were chosen via a simple grid search on a validation set. All the results are reported on the test set. The classification results of all considered models are shown in Table 1. We also compare our model trained on the original training data and one augmented by naively adding the 3 M matched pairs. For confidentiality reasons, we report the performance relative to some baseline. We see from Table 1 that purely augmenting the training data with the matched samples does not improve but worsens the base classifier. Table 2 shows the performance of the QUARTS compared with MatchPyramid models and the DSSM model initialized with pretrained BERT embeddings. The end-to-end QUARTS model beats the BERT DSSM baseline by over 17% in APR, and over 26% in F-score. To validate the effectiveness of QUARTS in improving the ranking performance for the search task, we performed an A/B test on live search traffic in two countries, to account for varying traffic patterns. Compared to the existing baselines, the QUARTS model yielded a 12 . 2% and 5 . 75% increase in online metrics for the two countries respectively, which are significant given the task. We used a held-out 10% of the ( I, Q, Q gen ) data to evaluate the VED generator. In order to make a fair evaluation, we ensured that the items that appeared in training set were not in the validation set. The validation BLEU scores are shown in Table 4. BLEU scores do not indicate whether or not a generated queries is a realistic modification of the original query. Therefore, we also had 2500 generated pairs annotated by human experts who were specifically trained to decide if a query-item Item title ( I ) Query ( Q ) Generated query ( Q gen ) ESR iPhone 8/7 screen protector tempered glass... iPhone 8 curved screen protector iPhone 8 plus cases JETech case for iPad Pro 12.9 inch ipad pro 12.9 speck shell iPad pro 12.9 Mounting dream full motion wall mounts bracket lg oled tv mount 55 inch flat screen tv Intel core i7-8700K desktop processor 6 cores core i7 8700k GTX 1080 Chicco pocket snack booster seat peg perego high chair baby dining set Comfy sheets ultra luxury 100% Egyptian cotton sheet set king size sheets king size beds for sale Table 3: Examples of adversarial query generations from the VED query generator. pair is matched or not. The accuracy 82% in Table 4 suggests that most of generated pairs are meaningful. Here, the accuracy is the fraction of the pairs that were actually labeled as mismatches Model BLEU-1 BLEU-2 BLEU-3 BLEU-4 Acc VED 35.15 31.40 24.84 20.76 0.82 Table 4: Validation BLEU scores of generated queries from the variational encoder-decoder generator, and misclassification accuracy as reported by humans. We provide some qualitative results from the VED in Table 3. The generator's goal is to slightly modify the input query Q , so that the resultant ( I, Q gen ) sample is realistic. A source query for screen protector is mapped to a query for phone case , and a source query for tv mount is mapped to one for flat screen tv . The goal of the word-by-word attention layer is to understand what parts of the user query and item titles are important to understand whether to match or not. Importantly, item titles are typically long, and have information such as brand, color and size. All of these facets might not be relevant for a particular user query. Figure 3 shows the performance of the word-by-word attention layer, for a matched and a mismatched pair. In both cases, we see that the correct words are attended to, helping the classifier make the distinction between a matched and a mismatched pair. Figure 4 shows another example. We developed an end-to-end model with hard to classify query generation for retrieval in e-commerce product search. We built upon ideas for textual entailment, and used a word by word attention layer to help create item representations conditioned on an input query. We trained a generator that yields representations of queries that are mismatched to a source item, while at the same time being realistic. This allows us to address the Figure 3: Word-by-word attention for a mismatched (top) and matched (bottom) query-item pair. To train the model end to end, we modified the cross-entropy loss, allowing us to avoid optimizing a minimax objective. Experiments on an offline dataset and live product search traffic showed that our method improves significantly over baselines.
abstain abstain abstain objective method abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain result method objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain result result abstain abstain method abstain objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain other abstain abstain method abstain method abstain method method abstain method method other abstain abstain abstain method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method method abstain method abstain method method method method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective method method abstain objective result