Why Not to Use Zero Imputation? Correcting Sparsity Bias in Training Neural Networks
Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that it adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.
This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input. The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method. In general I think the paper is descent and elegant. It is motivated from real-world pain-point, gives a rigorous study towards the root cause, and the proposed method is very effective. To the best of my knowledge there is no prior work looking deep into this area and this paper does bring new insights to the community. As a result I would vote for its acceptance. One issue is that I find the backbone methods in experiments are somehow out-of-date. For example, AutoRec (2015) and CF-NADE (2016). I admit that I’m not an expert in the field of recommendation but still think that more recent, and powerful baseline algorithms should be applied on to further demonstrate the true effectiveness of Sparsity Normalization.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Why Not to Use Zero Imputation? Correcting Sparsity Bias in Training Neural Networks
Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal performances in training neural networks. Yet, none of the existing work has explained what brings such performance degradations. In this paper, we introduce the variable sparsity problem (VSP), which describes a phenomenon where the output of a predictive model largely varies with respect to the rate of missingness in the given input, and show that it adversarially affects the model performance. We first theoretically analyze this phenomenon and propose a simple yet effective technique to handle missingness, which we refer to as Sparsity Normalization (SN), that directly targets and resolves the VSP. We further experimentally validate SN on diverse benchmark datasets, to show that debiasing the effect of input-level sparsity improves the performance and stabilizes the training of neural networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input. The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method. In general I think the paper is descent and elegant. It is motivated from real-world pain-point, gives a rigorous study towards the root cause, and the proposed method is very effective. To the best of my knowledge there is no prior work looking deep into this area and this paper does bring new insights to the community. As a result I would vote for its acceptance. One issue is that I find the backbone methods in experiments are somehow out-of-date. For example, AutoRec (2015) and CF-NADE (2016). I admit that I’m not an expert in the field of recommendation but still think that more recent, and powerful baseline algorithms should be applied on to further demonstrate the true effectiveness of Sparsity Normalization.
assistant
This paper studies a very interesting phenomena in machine learning called VSP, that is the output of the model is highly affected via the level of missing values in its input. The authors demonstrate the existence of such phenomena empirically, analyze the root cause for it theoretically, and propose a simple yet effective normalization method to tackle the problem. Several experiments demonstrate the effectiveness of this method. In general I think the paper is descent and elegant. It is motivated from real-world pain-point, gives a rigorous study towards the root cause, and the proposed method is very effective. To the best of my knowledge there is no prior work looking deep into this area and this paper does bring new insights to the community. As a result I would vote for its acceptance. One issue is that I find the backbone methods in experiments are somehow out-of-date. For example, AutoRec (2015) and CF-NADE (2016). I admit that I’m not an expert in the field of recommendation but still think that more recent, and powerful baseline algorithms should be applied on to further demonstrate the true effectiveness of Sparsity Normalization.