Neural Networks (NN) have achieved state-of-the-art performance in many tasks within image, speech, and text domains. Such great success is mainly due to special structure design to fit the particular data patterns, such as CNN capturing spatial locality and RNN modeling sequential dependency. Essentially, these specific NNs achieve good performance by leveraging the prior knowledge over corresponding domain data. Nevertheless, there are many applications with all kinds of tabular data in other domains. Since there are no shared patterns among these diverse tabular data, it is hard to design specific structures to fit them all. Without careful architecture design based on domain knowledge, it is quite challenging for NN to reach satisfactory performance in these tabular data domains. To fill the gap of NN in tabular data learning, we propose a universal neural network solution, called TabNN, to derive effective NN architectures for tabular data in all kinds of tasks automatically. Specifically, the design of TabNN follows two principles: to explicitly leverage expressive feature combinations and to reduce model complexity. Since GBDT has empirically proven its strength in modeling tabular data, we use GBDT to power the implementation of TabNN. Comprehensive experimental analysis on a variety of tabular datasets demonstrate that TabNN can achieve much better performance than many baseline solutions.
This paper proposes a hybrid machine learning algorithm using Gradient Boosted Decision Trees (GBDT) and Deep Neural Networks (DNN). The intended research direction on tabular data is essential and promising. However, the proposed technique does not seem to be handling the problem foundationally well. It seems heavily dependent on GBDT. It also shows itself in the results that final algorithm is almost indistinguishable from GBDT regarding results. Moreover, I  don't think that the data sets in experiments are good enough to cover the importance and the nature of the problem. 
Pros:
-This is a crucial line of research direction that aims to make DNNs applicable to many real-world problems (beyond speech and vision) in which discrete data and heterogeneous features exist such as engagement prediction, recommendation, and search.  
-The starting point of using GBDT seems like a good choice. 
-The Paper is mostly well written except occasional repetitions and missing acronym definitions.
Cons:
-The proposed technique does not seem to be original enough, and it does not handle the problem foundationally well. I do not think that there is enough justification/demonstration for the fact that a general NN solution for Tabular Data invented. The proposed technique is heavily dependent on GBDT (Indeed the algorithm and the learned trees are used at least three times). This shows itself in the results; i.e., the proposed algorithm is either negligibly performing better than GBDT or when  GBDT dependence removed, it performs worse. It seems to me that (except the minor small section of streaming data), the paper is more like a proper verification of how tree-based learning algorithms work very well in tabular data--which is far from the basis of the paper and does not make the paper novel enough for ICLR.   
-The proposed technique seems to include very heavy feature engineering and several ad-hoc practical steps--that is far from the motivation of using NN in tabular data.
-In the provided benchmark data sets the depth of the analysis seems to be enough. However, in the proposed domain of tabular data, often data sets are significantly more high dimensional in reality and include at least one set of sparse large dimensional features  (e.g., unstructured raw text for the search queries.) In such scenarios, it had been showed that wide-and-deep NNs perform decently. However such problems are entirely missing in the results section. I also think that this is a lost opportunity for the authors as they could be showing that it is the NN part contributing.