We propose a novel high-performance interpretable deep tabular data learning network, TabNet. TabNet utilizes a sequential attention mechanism that softly selects features to reason from at each decision step and then aggregates the processed information to make a final prediction decision. By explicitly selecting sparse features, TabNet learns very efficiently as the model capacity at each decision step is fully utilized for the most relevant features, resulting in a high performance model. This sparsity also enables more interpretable decision making through the visualization of feature selection masks. We demonstrate that TabNet outperforms other neural network and decision tree variants on a wide range of tabular data learning datasets, especially those are not saturated in performance, and yields interpretable feature attributions and insights into the global model behavior.
This paper proposes a new neural network for tabular data which uses sequential attention to perform instance-wise feature selection which can help learn superior decision rules, in addition to facilitating interpretability of the resulting model.  Overall, the paper is well-written and fairly easy to follow, and the idea appears conceptually well-grounded and of high practical value. However, I have concerns about the experiments.
As the proposed model is primarily empirically motivated (ie. there is no theoretical justification for why it should be better), it is imperative the authors conduct thorough experiments to convincingly demonstrate its utility.  However, the current experiments are too sparse to be fully convincing, given how easy it is nowadays to evaluate ML models across diverse collections of tabular datasets (eg. OpenML, UCI, CatBoost). While other papers on tabular-data models study many real datasets (eg. Klambauer et al), this paper only studies 4 real datasets (and does not even quantify the variability in results across multiple training/test splits). 
Also, why did the authors not compare against xgboost/lightGBM/catboost, some of the most popular tabular models, in all datasets? 
It seems different hyperparameter values were also used for each dataset, what was the search-space / hyperparameter optimization method used here? Or were hyperparameters for each dataset found through manual tuning.
Klambauer et al (2017). Self-Normalizing Neural Networks. 
https://arxiv.org/abs/1706.02515
OpenML: https://www.openml.org
UCI: https://archive.ics.uci.edu/ml/datasets.php
CatBoost: https://catboost.ai/#benchmark
Other comments:
- Why don't the authors quantitatively compare the quality of the selected features from their model vs other feature selection methods for the Syn datasets?
- What is "softmax-training" in Appendix D? If the authors just mean standard training, I recommend they use this name instead.
- Since there are many components to this model, it would help clarify their purpose to the reader if the authors could provide one concrete example of the feedforward pass of a hypothetical instantiation of their model, say for a simple datapoint with just 2 features.
- the subscript i in h_i should match the i used in M[i], P[i] to avoid confusion
- P[0] (the base case) needs to be explained.
- a[i-1] should be more clearly defined, and the base case explained in more detail (I assume it is just the input features). 
 
- I think it is a bit confusing to present the model parameters in terms of batch-size B, and think it would be better if you introduced them assuming batch-size = 1 (the generalization to batch-size B is obvious and should be familiar to readers as it's the same for all neural models).