NA
NA
 The most important criticism I have is about the claim that the marginal increase in parameters is negligible. I think it*s _probably_ true, but given that the increases in performance are quite small across the board (<0.5 gain in accuracy), and that all of the learned thresholds are so close to 0 (i.e. not much different from pure relu), I think it would have been important and pretty easy just to remove a few thousand parameters from the convrelu architecture to equalize the number of parameters in each network. I think would need to see accuracies for at least one experiment like this to increase my rating, or to be somehow otherwise convinced that the number of parameters does not make the difference. The background and related work section covers most of the recent ReLU alternatives, but doesn*t not clearly explain what an activation function is, and misses some references. I find the way the biological motivation is presented to be a bit misleading/overstated. It*s fine and interesting to say that the idea was inspired by something about biological neurons, but I think it*s a stretch to claim to "mimic brain function". The related work section is pretty good, but reads a bit like a list of papers - it would be nice to have a bit more discussion of how related methods are similar/different to the proposed method. The experimental sections, tables, and plot of learned threshold values are good, but impression from the swish paper and others was that ReLU variants do not perform much differently on smaller datasets, and only make a difference in very deep networks. If my impression is correct, it would be important to do an experiment with a deep resnet or something like that, but I*m not very sure about this impression. The exploration of the partial replacement strategy is interesting and worthwhile, and again might be even more interesting in much deeper networks. Most hyperparameter settings are mentioned, but what about the learning rate? Was this tuned for each experiment? I seem to remember there being a paper that showed learning rate made a pretty big difference for different activation functions* performance. The tinyimagenet numbers seem a little off; I don*t have personal recent experience with this dataset but this blog post claims 56.4 with normal ReLUs in a VGGish architecture. https://learningai.io/projects/2017/06/29/tiny-imagenet.html Some other questions: - How similar are the distributions of learned thresholds across datasets? - How much do the thresholds change over the course of training (do they quickly reach the values shown in the histogram and stay that way, or are they very noisy? - An interesting experiment could be to use fixed thresholds according to the distribution shown in Fig. 6. Especially if the distributions are similar across datasets, it may be that this performs as well as learning the thresholds, without introducing extra parameters to learn. This would also help disentangle the effect of having different thresholds vs. the possible implicit regularization from the noise of the thresholds changing. Other refs to mention: Randomized relu (randomly set the leak threshold) https://arxiv.org/pdf/1505.00853.pdf Noisy activation functions Quality: Decent (6.5/10), but I think the experiments, level of analysis, and quality of writing are more like a very good blog post than an academic paper (although I think that that line is becoming very blurry) Clarity: Decent (7/10), but there are many small grammar errors, especially to do with pluralization (e.g. "activation functions .... its importance" should be "their importance") and incorrect verb tense (e.g. "was used" should be "is used" or "has been used"; past tense implies it no longer is used). I pointed out the ones that were particularly confusing in the "specific comments" section, but the paper should be thoroughly reviewed for writing quality. The idea is simple and its implementation and the experiments are well explained. Originality: (6.5/10) As far as I know the idea of parameterized thresholds for relus is original, as is the partial replacement strategy, but I am not an expert in this area. There is not much/any originality or creativity in the analysis of the results or probing of why variable thresholds are better; I think this could make the paper much stronger. Significance: (5.5/10) For anyone looking to squeeze a little extra performance out of a model for some reason, I think even the <0.5 improvement is useful to know about. This seems more like an engineering problem than a research problem though, and I think that the paper is lacking in significant research insight / exploration. The gains might be more significant in deeper networks, but this is difficult to assess without experiments. Pros: - marginal improvement across the board - partial replacement strategy might be useful for other methods Cons: - Writing contains grammatical errors / change of tense / confusing sentence construction which make the paper somewhat hard to get through - Not much exploration or insight about the learned functions or the activation function performs well - Numbers seem quite incremental to me, and the method is not tested on larger/deeper networks where the gains might be mode significant. Specific comments/nits (in order reading through paper): 1. "ReLUs can solve the gradient vanishing and accelerate training convergence" <- cite this claim 2. it suffers -> they suffer 3. "suffer from the dying relu problem" <- cite this claim 4. "For the parameters involved in ..." this sentence makes it sound like you don*t tryi anything other than L2, and it does not explain why you want the trainable thresholds close to zero. 5. "we propose a ...method known as adaptive ReLUs" -> "method we call adaptive ReLUs" or something like that (it can*t be known as it yet; you*ve just proposed it). 6. "due to its importance in deep neural networks" this is not an explanation of _why_ activations are a popular research field; it just raises the further question of why they are important in DNNs. 7. tanh: describe why the zerocentering property is good if you want to mention it being preferred for that reason 8. mention that motivation for elu etc. is to be soft/differentiable 9. "the use in convolution is not clear" what do you mean by this? that it hasn*t worked well in practice or that they don*t work by construction for some reason... explain 10. Not clear what the swish function is or how it is related to ELUs and SELUs. 11. "well known that neural networks mimic computational activities in the brain" I would not say this is well known. It*s a contentious claim; "inspired by" or "modeled after" would be more accurate in my opinion. 12. "which is also called" this is confusing/misleadingly worded; the process of the brain inputting and outputting signals is NOT "also called feedforward network with relus" 13. "which can better mimic brain functions" most would consider neuron firing is a process that occurs in the brain, not a "brain function" which would be something like "seeing" 14. "share the same outgoing weight" this is confusing; makes it sound like you dynamically check the value of the weight and group units based on this value. 15. "Glot initialization" -> "Glorot initialization" 16. "zero init was used" -> "has been used" 17. Cite densenet in Fig.4 caption 18. "with minor adjustments to accomodate diferent datasets" "standard data augmentation" explain what these are (in appendix if necessary) 19. "VGG like network" cite VGG and explain what is different about yours
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
NA
NA
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The most important criticism I have is about the claim that the marginal increase in parameters is negligible. I think it*s _probably_ true, but given that the increases in performance are quite small across the board (<0.5 gain in accuracy), and that all of the learned thresholds are so close to 0 (i.e. not much different from pure relu), I think it would have been important and pretty easy just to remove a few thousand parameters from the convrelu architecture to equalize the number of parameters in each network. I think would need to see accuracies for at least one experiment like this to increase my rating, or to be somehow otherwise convinced that the number of parameters does not make the difference. The background and related work section covers most of the recent ReLU alternatives, but doesn*t not clearly explain what an activation function is, and misses some references. I find the way the biological motivation is presented to be a bit misleading/overstated. It*s fine and interesting to say that the idea was inspired by something about biological neurons, but I think it*s a stretch to claim to "mimic brain function". The related work section is pretty good, but reads a bit like a list of papers - it would be nice to have a bit more discussion of how related methods are similar/different to the proposed method. The experimental sections, tables, and plot of learned threshold values are good, but impression from the swish paper and others was that ReLU variants do not perform much differently on smaller datasets, and only make a difference in very deep networks. If my impression is correct, it would be important to do an experiment with a deep resnet or something like that, but I*m not very sure about this impression. The exploration of the partial replacement strategy is interesting and worthwhile, and again might be even more interesting in much deeper networks. Most hyperparameter settings are mentioned, but what about the learning rate? Was this tuned for each experiment? I seem to remember there being a paper that showed learning rate made a pretty big difference for different activation functions* performance. The tinyimagenet numbers seem a little off; I don*t have personal recent experience with this dataset but this blog post claims 56.4 with normal ReLUs in a VGGish architecture. https://learningai.io/projects/2017/06/29/tiny-imagenet.html Some other questions: - How similar are the distributions of learned thresholds across datasets? - How much do the thresholds change over the course of training (do they quickly reach the values shown in the histogram and stay that way, or are they very noisy? - An interesting experiment could be to use fixed thresholds according to the distribution shown in Fig. 6. Especially if the distributions are similar across datasets, it may be that this performs as well as learning the thresholds, without introducing extra parameters to learn. This would also help disentangle the effect of having different thresholds vs. the possible implicit regularization from the noise of the thresholds changing. Other refs to mention: Randomized relu (randomly set the leak threshold) https://arxiv.org/pdf/1505.00853.pdf Noisy activation functions Quality: Decent (6.5/10), but I think the experiments, level of analysis, and quality of writing are more like a very good blog post than an academic paper (although I think that that line is becoming very blurry) Clarity: Decent (7/10), but there are many small grammar errors, especially to do with pluralization (e.g. "activation functions .... its importance" should be "their importance") and incorrect verb tense (e.g. "was used" should be "is used" or "has been used"; past tense implies it no longer is used). I pointed out the ones that were particularly confusing in the "specific comments" section, but the paper should be thoroughly reviewed for writing quality. The idea is simple and its implementation and the experiments are well explained. Originality: (6.5/10) As far as I know the idea of parameterized thresholds for relus is original, as is the partial replacement strategy, but I am not an expert in this area. There is not much/any originality or creativity in the analysis of the results or probing of why variable thresholds are better; I think this could make the paper much stronger. Significance: (5.5/10) For anyone looking to squeeze a little extra performance out of a model for some reason, I think even the <0.5 improvement is useful to know about. This seems more like an engineering problem than a research problem though, and I think that the paper is lacking in significant research insight / exploration. The gains might be more significant in deeper networks, but this is difficult to assess without experiments. Pros: - marginal improvement across the board - partial replacement strategy might be useful for other methods Cons: - Writing contains grammatical errors / change of tense / confusing sentence construction which make the paper somewhat hard to get through - Not much exploration or insight about the learned functions or the activation function performs well - Numbers seem quite incremental to me, and the method is not tested on larger/deeper networks where the gains might be mode significant. Specific comments/nits (in order reading through paper): 1. "ReLUs can solve the gradient vanishing and accelerate training convergence" <- cite this claim 2. it suffers -> they suffer 3. "suffer from the dying relu problem" <- cite this claim 4. "For the parameters involved in ..." this sentence makes it sound like you don*t tryi anything other than L2, and it does not explain why you want the trainable thresholds close to zero. 5. "we propose a ...method known as adaptive ReLUs" -> "method we call adaptive ReLUs" or something like that (it can*t be known as it yet; you*ve just proposed it). 6. "due to its importance in deep neural networks" this is not an explanation of _why_ activations are a popular research field; it just raises the further question of why they are important in DNNs. 7. tanh: describe why the zerocentering property is good if you want to mention it being preferred for that reason 8. mention that motivation for elu etc. is to be soft/differentiable 9. "the use in convolution is not clear" what do you mean by this? that it hasn*t worked well in practice or that they don*t work by construction for some reason... explain 10. Not clear what the swish function is or how it is related to ELUs and SELUs. 11. "well known that neural networks mimic computational activities in the brain" I would not say this is well known. It*s a contentious claim; "inspired by" or "modeled after" would be more accurate in my opinion. 12. "which is also called" this is confusing/misleadingly worded; the process of the brain inputting and outputting signals is NOT "also called feedforward network with relus" 13. "which can better mimic brain functions" most would consider neuron firing is a process that occurs in the brain, not a "brain function" which would be something like "seeing" 14. "share the same outgoing weight" this is confusing; makes it sound like you dynamically check the value of the weight and group units based on this value. 15. "Glot initialization" -> "Glorot initialization" 16. "zero init was used" -> "has been used" 17. Cite densenet in Fig.4 caption 18. "with minor adjustments to accomodate diferent datasets" "standard data augmentation" explain what these are (in appendix if necessary) 19. "VGG like network" cite VGG and explain what is different about yours
assistant
 The most important criticism I have is about the claim that the marginal increase in parameters is negligible. I think it*s _probably_ true, but given that the increases in performance are quite small across the board (<0.5 gain in accuracy), and that all of the learned thresholds are so close to 0 (i.e. not much different from pure relu), I think it would have been important and pretty easy just to remove a few thousand parameters from the convrelu architecture to equalize the number of parameters in each network. I think would need to see accuracies for at least one experiment like this to increase my rating, or to be somehow otherwise convinced that the number of parameters does not make the difference. The background and related work section covers most of the recent ReLU alternatives, but doesn*t not clearly explain what an activation function is, and misses some references. I find the way the biological motivation is presented to be a bit misleading/overstated. It*s fine and interesting to say that the idea was inspired by something about biological neurons, but I think it*s a stretch to claim to "mimic brain function". The related work section is pretty good, but reads a bit like a list of papers - it would be nice to have a bit more discussion of how related methods are similar/different to the proposed method. The experimental sections, tables, and plot of learned threshold values are good, but impression from the swish paper and others was that ReLU variants do not perform much differently on smaller datasets, and only make a difference in very deep networks. If my impression is correct, it would be important to do an experiment with a deep resnet or something like that, but I*m not very sure about this impression. The exploration of the partial replacement strategy is interesting and worthwhile, and again might be even more interesting in much deeper networks. Most hyperparameter settings are mentioned, but what about the learning rate? Was this tuned for each experiment? I seem to remember there being a paper that showed learning rate made a pretty big difference for different activation functions* performance. The tinyimagenet numbers seem a little off; I don*t have personal recent experience with this dataset but this blog post claims 56.4 with normal ReLUs in a VGGish architecture. https://learningai.io/projects/2017/06/29/tiny-imagenet.html Some other questions: - How similar are the distributions of learned thresholds across datasets? - How much do the thresholds change over the course of training (do they quickly reach the values shown in the histogram and stay that way, or are they very noisy? - An interesting experiment could be to use fixed thresholds according to the distribution shown in Fig. 6. Especially if the distributions are similar across datasets, it may be that this performs as well as learning the thresholds, without introducing extra parameters to learn. This would also help disentangle the effect of having different thresholds vs. the possible implicit regularization from the noise of the thresholds changing. Other refs to mention: Randomized relu (randomly set the leak threshold) https://arxiv.org/pdf/1505.00853.pdf Noisy activation functions Quality: Decent (6.5/10), but I think the experiments, level of analysis, and quality of writing are more like a very good blog post than an academic paper (although I think that that line is becoming very blurry) Clarity: Decent (7/10), but there are many small grammar errors, especially to do with pluralization (e.g. "activation functions .... its importance" should be "their importance") and incorrect verb tense (e.g. "was used" should be "is used" or "has been used"; past tense implies it no longer is used). I pointed out the ones that were particularly confusing in the "specific comments" section, but the paper should be thoroughly reviewed for writing quality. The idea is simple and its implementation and the experiments are well explained. Originality: (6.5/10) As far as I know the idea of parameterized thresholds for relus is original, as is the partial replacement strategy, but I am not an expert in this area. There is not much/any originality or creativity in the analysis of the results or probing of why variable thresholds are better; I think this could make the paper much stronger. Significance: (5.5/10) For anyone looking to squeeze a little extra performance out of a model for some reason, I think even the <0.5 improvement is useful to know about. This seems more like an engineering problem than a research problem though, and I think that the paper is lacking in significant research insight / exploration. The gains might be more significant in deeper networks, but this is difficult to assess without experiments. Pros: - marginal improvement across the board - partial replacement strategy might be useful for other methods Cons: - Writing contains grammatical errors / change of tense / confusing sentence construction which make the paper somewhat hard to get through - Not much exploration or insight about the learned functions or the activation function performs well - Numbers seem quite incremental to me, and the method is not tested on larger/deeper networks where the gains might be mode significant. Specific comments/nits (in order reading through paper): 1. "ReLUs can solve the gradient vanishing and accelerate training convergence" <- cite this claim 2. it suffers -> they suffer 3. "suffer from the dying relu problem" <- cite this claim 4. "For the parameters involved in ..." this sentence makes it sound like you don*t tryi anything other than L2, and it does not explain why you want the trainable thresholds close to zero. 5. "we propose a ...method known as adaptive ReLUs" -> "method we call adaptive ReLUs" or something like that (it can*t be known as it yet; you*ve just proposed it). 6. "due to its importance in deep neural networks" this is not an explanation of _why_ activations are a popular research field; it just raises the further question of why they are important in DNNs. 7. tanh: describe why the zerocentering property is good if you want to mention it being preferred for that reason 8. mention that motivation for elu etc. is to be soft/differentiable 9. "the use in convolution is not clear" what do you mean by this? that it hasn*t worked well in practice or that they don*t work by construction for some reason... explain 10. Not clear what the swish function is or how it is related to ELUs and SELUs. 11. "well known that neural networks mimic computational activities in the brain" I would not say this is well known. It*s a contentious claim; "inspired by" or "modeled after" would be more accurate in my opinion. 12. "which is also called" this is confusing/misleadingly worded; the process of the brain inputting and outputting signals is NOT "also called feedforward network with relus" 13. "which can better mimic brain functions" most would consider neuron firing is a process that occurs in the brain, not a "brain function" which would be something like "seeing" 14. "share the same outgoing weight" this is confusing; makes it sound like you dynamically check the value of the weight and group units based on this value. 15. "Glot initialization" -> "Glorot initialization" 16. "zero init was used" -> "has been used" 17. Cite densenet in Fig.4 caption 18. "with minor adjustments to accomodate diferent datasets" "standard data augmentation" explain what these are (in appendix if necessary) 19. "VGG like network" cite VGG and explain what is different about yours