Utility Analysis of Network Architectures for 3D Point Cloud Processing
In this paper, we diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures. We propose a number of hypotheses on the effects of specific network architectures on the representation capacity of DNNs. In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency. We conduct comparative studies based on such metrics to verify the hypotheses, which may shed new lights on the architectural design of neural networks. Experiments demonstrated the effectiveness of our method. The code will be released when this paper is accepted.
The paper addresses the task of evaluation of point-based architectures for deep learning, aiming at establishing the contribution of various architectural blocks (e.g., local density reweighting) to a number of desirable properties of the resulting models (such as adversarial robustness, rotation robustness, etc.). To assess the influence of such blocks on the resulting models, authors formulate and test a variety hypotheses while measuring model performance using a number of existing measures, such as information discarding/concentration, etc. I believe the paper cannot be accepted in its present form, as (1) no clear motivation for the work is presented, (2) the problem statement and the research questions are very blurry, and consequently (3) the results do not provide significant new knowledge related to the point-based neural architectures. While the specific architectural choices do influence the characteristics of commonly used models and a systematic evaluation of the former is indeed useful for practice, the paper does not indicate a particular application context, which I believe is crucial to the entire following narrative. Do the authors aim at evaluating in the recognition/labeling context? In the synthesis context (e.g., novel shape synthesis, completion, or upsampling)? In the geometry processing context (normal estimation, patch decomposition, or denoising)? Would the claimed hypothesis, let alone the results, even be formulated the same way, once one switches to a different class of applications? If yes, it would be hard to understand from the paper. If no, this could potentially diminish the value of the entire work significantly. If the paper targeted a particular class of applications or simply a specific task, the hypotheses might have looked more natural; however, currently I find the motivation for these hypotheses rather vague. New insights into neural architectures or other models could emerge from (empirical and/or theoretical) regularities revealed by a purposeful experiment; however, the authors perform a seemingly random evaluation of point-based architectures which is of a fairly limited value per se. The reader might wonder: what is the exact problem that the study aims to address? Thus, I believe, a clarification of the problem statement and the basic research questions for the entire work is needed. The paper does a good job of thoroughly presenting results of numerous computational experiments and values of the selected measures. However, as stated before, the results fail to provide significant insights into point-based neural architectures due to (1) lack of specific applications the authors are trying to solve and (2) arbitrariness of the chosen experiments lead to equally arbitrary conclusions. If “architecture of We et al., which uses local density information, improves adversarial robustness”, what difference would it make? Would it open novel opportunities for point-based deep nets, and if yes, which exactly? One minor additions is that since the paper focuses on empirical evaluation of machine learning models, significance measures such a statistical confidence intervals would be very useful to present in the results.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Utility Analysis of Network Architectures for 3D Point Cloud Processing
In this paper, we diagnose deep neural networks for 3D point cloud processing to explore the utility of different network architectures. We propose a number of hypotheses on the effects of specific network architectures on the representation capacity of DNNs. In order to prove the hypotheses, we design five metrics to diagnose various types of DNNs from the following perspectives, information discarding, information concentration, rotation robustness, adversarial robustness, and neighborhood inconsistency. We conduct comparative studies based on such metrics to verify the hypotheses, which may shed new lights on the architectural design of neural networks. Experiments demonstrated the effectiveness of our method. The code will be released when this paper is accepted.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper addresses the task of evaluation of point-based architectures for deep learning, aiming at establishing the contribution of various architectural blocks (e.g., local density reweighting) to a number of desirable properties of the resulting models (such as adversarial robustness, rotation robustness, etc.). To assess the influence of such blocks on the resulting models, authors formulate and test a variety hypotheses while measuring model performance using a number of existing measures, such as information discarding/concentration, etc. I believe the paper cannot be accepted in its present form, as (1) no clear motivation for the work is presented, (2) the problem statement and the research questions are very blurry, and consequently (3) the results do not provide significant new knowledge related to the point-based neural architectures. While the specific architectural choices do influence the characteristics of commonly used models and a systematic evaluation of the former is indeed useful for practice, the paper does not indicate a particular application context, which I believe is crucial to the entire following narrative. Do the authors aim at evaluating in the recognition/labeling context? In the synthesis context (e.g., novel shape synthesis, completion, or upsampling)? In the geometry processing context (normal estimation, patch decomposition, or denoising)? Would the claimed hypothesis, let alone the results, even be formulated the same way, once one switches to a different class of applications? If yes, it would be hard to understand from the paper. If no, this could potentially diminish the value of the entire work significantly. If the paper targeted a particular class of applications or simply a specific task, the hypotheses might have looked more natural; however, currently I find the motivation for these hypotheses rather vague. New insights into neural architectures or other models could emerge from (empirical and/or theoretical) regularities revealed by a purposeful experiment; however, the authors perform a seemingly random evaluation of point-based architectures which is of a fairly limited value per se. The reader might wonder: what is the exact problem that the study aims to address? Thus, I believe, a clarification of the problem statement and the basic research questions for the entire work is needed. The paper does a good job of thoroughly presenting results of numerous computational experiments and values of the selected measures. However, as stated before, the results fail to provide significant insights into point-based neural architectures due to (1) lack of specific applications the authors are trying to solve and (2) arbitrariness of the chosen experiments lead to equally arbitrary conclusions. If “architecture of We et al., which uses local density information, improves adversarial robustness”, what difference would it make? Would it open novel opportunities for point-based deep nets, and if yes, which exactly? One minor additions is that since the paper focuses on empirical evaluation of machine learning models, significance measures such a statistical confidence intervals would be very useful to present in the results.
assistant
The paper addresses the task of evaluation of point-based architectures for deep learning, aiming at establishing the contribution of various architectural blocks (e.g., local density reweighting) to a number of desirable properties of the resulting models (such as adversarial robustness, rotation robustness, etc.). To assess the influence of such blocks on the resulting models, authors formulate and test a variety hypotheses while measuring model performance using a number of existing measures, such as information discarding/concentration, etc. I believe the paper cannot be accepted in its present form, as (1) no clear motivation for the work is presented, (2) the problem statement and the research questions are very blurry, and consequently (3) the results do not provide significant new knowledge related to the point-based neural architectures. While the specific architectural choices do influence the characteristics of commonly used models and a systematic evaluation of the former is indeed useful for practice, the paper does not indicate a particular application context, which I believe is crucial to the entire following narrative. Do the authors aim at evaluating in the recognition/labeling context? In the synthesis context (e.g., novel shape synthesis, completion, or upsampling)? In the geometry processing context (normal estimation, patch decomposition, or denoising)? Would the claimed hypothesis, let alone the results, even be formulated the same way, once one switches to a different class of applications? If yes, it would be hard to understand from the paper. If no, this could potentially diminish the value of the entire work significantly. If the paper targeted a particular class of applications or simply a specific task, the hypotheses might have looked more natural; however, currently I find the motivation for these hypotheses rather vague. New insights into neural architectures or other models could emerge from (empirical and/or theoretical) regularities revealed by a purposeful experiment; however, the authors perform a seemingly random evaluation of point-based architectures which is of a fairly limited value per se. The reader might wonder: what is the exact problem that the study aims to address? Thus, I believe, a clarification of the problem statement and the basic research questions for the entire work is needed. The paper does a good job of thoroughly presenting results of numerous computational experiments and values of the selected measures. However, as stated before, the results fail to provide significant insights into point-based neural architectures due to (1) lack of specific applications the authors are trying to solve and (2) arbitrariness of the chosen experiments lead to equally arbitrary conclusions. If “architecture of We et al., which uses local density information, improves adversarial robustness”, what difference would it make? Would it open novel opportunities for point-based deep nets, and if yes, which exactly? One minor additions is that since the paper focuses on empirical evaluation of machine learning models, significance measures such a statistical confidence intervals would be very useful to present in the results.