Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that it adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.
This paper provides a novel solution to the variable sparsity problem, where the output of neural networks biased with respect to the number of missing inputs. The authors proposed a sparsity normalization algorithm to process the input vectors to encounter the bias. In experiments, the authors evaluated the proposed sparsity normalization model on multiple datasets: collaborative filtering datasets, electric medical records datasets, single-cell RNA sequence datasets and UCI datasets. Results show that the proposed normalization method improves the prediction performance and the predicted values of the neural network is more uniformly distributed according to the number of missing entries.
The paper describes a clear and specific machine learning problem. Then the authors demonstrate a simple normalization strategy is capable of fixing the issue of biased prediction. The paper has a well-organized structure to convey the motivation. Therefore, my opinion on this paper leans to an acceptation. My questions are mainly on the experiment section:
1) As shown in Table 2, there are various new collaborative filtering methods proposed after 2015, why the authors chose to extend AutoRec (Sedhain et al., 2015) but not other new methods?
2) In the experiments, you compare your model with zero imputation (Please correct me if w/o SN is not zero imputation). However, I think it is a common practice in machine learning that we perform imputation with mean or median values. I'm interested in knowing whether filling with mean/median values work with these datasets.
3) In section 4.5, you mentioned that "SN is effective
even when MCAR assumption is not established". However, I'm still not clear about the reason. I believe many machine learning datasets have NMAR (not missing at random) type of missing data, but not MCAR. So this is an important issue for me.
4) Does your model assume all input values are numerical but not categorical?