Self-attention has been widely used to model the sequential data and achieved remarkable results in many applications. Although it can be used to model dependencies without regard to positions of sequences, self-attention is seldom applied to non-sequential data. In this work, we propose to learn representations of multi-field categorical data in prediction tasks via self-attention mechanism, where features are orderless but have intrinsic relations over different fields. In most current DNN based models, feature embeddings are simply concatenated for further processing by networks. Instead, by applying self-attention to transform the embeddings, we are able to relate features in different fields and automatically learn representations of their combinations, which are known as the factors of many prevailing linear models. To further improve the effect of feature combination mining, we modify the original self-attention structure by restricting the similarity weight to have at most k non-zero values, which additionally regularizes the model. We experimentally evaluate the effectiveness of our self-attention model on non-sequential data. Across two click through rate prediction benchmark datasets, i.e., Cretio and Avazu, our model with top-k restricted self-attention achieves the state-of-the-art performance. Compared with the vanilla MLP, the gain by adding self-attention is significantly larger than that by modifying the network structures, which most current works focus on.
Quality: 
- In 4.4, the authors have vigorously explored the space of hyperparameters. However, they do not describe how to determine the hyperparameters, e.g., set aside a validation set from a part of the training set and determine the hyperparameters using this validation set, while the authors split the two datasets into only training and test sets, respectively. Without this procedure, the results may overfit to the test set via repeated experiments. Even though the used datasets are of few-million, this procedure guarantees a minimum requirement for a reliable outcome from the proposed model. I firmly recommend the authors to update their results using a validation set to determine the hyperparameters and then report on the test set. Please describe these experimental details to ensure that the performed experiments are valid.
   
Clarity:
- Overall, the writing can be improved via proof-reading and polishing the sentences. In Introduction section, "there is little work applying..." can be specified or rephrased with "it is underexplored to apply", and "input features are not independent" can be specified on what there are not independent. Moreover, the last two sentences in the second paragraph in the Introduction section is unclear what the authors want to argue: "The combinations in linear models are then made by cross product over different fields. Due to the sparsity problem, the combinations rely on much manual work of domain experts."
- The authors use top-k restriction (Shazeer et al., 2017) to consider sparse relationships among the features. For this reason, have you tried to use the L1 loss on the probability distributions, which are the outputs of softmax function?
- In 4.5, the authors said, they "are in most concern of complementarity." What is the reason for this idea and why not the "relevance"?
- In Table 4, I'm afraid that I don't understand the content (three numbers in parenthesis) of the third column. How does each input x_i or x_j, or a tuple of them get their own CTR?
Originality and significance:
- They apply self-attention to learn multiple categorical features to predict Click-Through-Rate (CTR) with a top-k non-zero similarity weight constraint to adapt to their categorical inputs. Due to this, the scientific contribution to the corresponding community is highly limited to providing empirical results on the CTR task.
- The authors argue that "most of current DNN-based models simply concatenate all feature embeddings"; however, this argument might be an over-simplified statement for the existing models in section 2.
- Similar works can be found but missed to cite: [1] proposes a general framework to self-attention to exploit sequential (time-domain) and parallel (feature-domain) non-locality. [2] learns bilinear attention maps to integrate multimodal inputs using skip-connections and multiple layers on top of the idea of low-rank bilinear pooling.
Pros:
- Strong empirical results on two CTR tasks using the previous works of self-attention and top-k restriction techniques.
Cons:
- This work fairly lacks its originality since the proposing method heavily relies on the two previous works, self-attention and top-k restriction. They apply them to multiple categorical features to estimate CTR; however, their application seems to be monotonic without a novel idea of task-specific adaptation.
Minor comments:
- In Figure 1, "the number of head" -> "the number of heads".
[1] Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local Neural Networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'18).
[2] Kim, J.-H., Jun, J., & Zhang, B.-T. (2018). Bilinear Attention Networks. In Advances in Neural Information Processing Systems 32 (NIPS'18).