Differential Equation Networks
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation networks, an improvement to modern neural networks in which each neuron learns the particular nonlinear activation function that it requires. We show that enabling each neuron with the ability to learn its own activation function results in a more compact network capable of achieving comperable, if not superior performance when compared to much larger networks. We also showcase the capability of a differential equation neuron to learn behaviors, such as oscillation, currently only obtainable by a large group of neurons. The ability of differential equation networks to essentially compress a large neural network, without loss of overall performance makes them suitable for on-device applications, where predictions must be computed locally. Our experimental evaluation of real-world and toy datasets show that differential equation networks outperform fixed activatoin networks in several areas.
 Personal Expertise: The reviewer has extensive practical and theoretical experience with deep networks, different activation functions, as well as some practical experience using deep networks to model physical phenomena which are different inverse problems than the more common perception modeling. However, the reviewer is not knowledgeable in using ODEs for deep networks. Contributions of the paper : The contribution of the paper is in proposing a learnable activation function in form of an ODE which can help to better model highly oscillatory and irregular functions more efficiently. This can be potentially useful for special applications in inverse problems where (by field knowledge) we know highly non-linear and specific activation functions can be more reasonable than the common ReLU and its recent variants. Quality and composition: The composition of the theoretical part of the paper is clear. The experimental part is very limited though and does not include all the necessary details. Many of the details in the initial sections can be taken to the appendices to make room for more empirical studies. Novelty, related works: The work seems novel in proposing the specific activation function but in general there are many other works that propose learnable or more elaborate activation functions, neurons, or local parts of networks. It is not so clear how this work is different from those and nor is compared to those works. This includes networks in networks, maxout networks, capsule networks, etc. Critique of the theories and experiments: Theoretical Design: The theoretical design and derivation of the paper seem correct, although the reviewer is not an expert on this topic. However, it does not clearly mention why the ODE is not designed and solved for each problem separately. Should there be different design choices for y for each task/dataset? Why not solving the ODE during the training as well? If we are solving the ODE only once and based on some initialization of coefficients, it seems to be equivalent to designing a learnable activation function such as leaky ReLU. In that regard, one could call leaky-ReLU a DifEN? Experimental Setup: The motivation of the new activation function is for specific use-cases where oscillatory or decaying functions are to be modelled. In that respect, the experimental setup is quite limited and inconclusive. - MNIST experiment: to conclusively evaluate the performance of the proposed activation function, it is important to try fixed activation functions on the same architecture as the DifEN and vice versa. - Diabetes regression experiment: since the task is not a well-studied regression task, more experiments on various datasets are required to make a conclusion. - The learnable activation function can potentially make the network more prone to overfitting, this needs to be tested thoroughly. - An important application of the proposed activation function is mentioned to be for model compression. That should be properly tested on problems with large sets of parameters (such as ImageNet networks) and observe if the performance drop due to a decrease in the number of parameters in a standard network is sharper than that of a network with DifEN activations. - More tasks and more analysis should be performed for the real-world tasks that the authors mention as the motivation of this work (specifically medical diagnosis or predictions). The analysis should demonstrate and give insight on the extra generalization power that the new activation function brings to those problems. - Following on the previous point, it is important to empirically demonstrate for which applications DifEN is useful. - Some analyses are missing on when a neuron can accentuate the problem of vanishing and/or exploding gradients in certain configurations of the DifEN parameters. It seems like the situation can arise during the training where the activation function become too steep or too saturated. Summary judgment: All in all, I think the proposed work has some potential in specific applications, however, the experimental setup does not give a clear and conclusive message of where and how the new activation functions are useful.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Differential Equation Networks
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation networks, an improvement to modern neural networks in which each neuron learns the particular nonlinear activation function that it requires. We show that enabling each neuron with the ability to learn its own activation function results in a more compact network capable of achieving comperable, if not superior performance when compared to much larger networks. We also showcase the capability of a differential equation neuron to learn behaviors, such as oscillation, currently only obtainable by a large group of neurons. The ability of differential equation networks to essentially compress a large neural network, without loss of overall performance makes them suitable for on-device applications, where predictions must be computed locally. Our experimental evaluation of real-world and toy datasets show that differential equation networks outperform fixed activatoin networks in several areas.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Personal Expertise: The reviewer has extensive practical and theoretical experience with deep networks, different activation functions, as well as some practical experience using deep networks to model physical phenomena which are different inverse problems than the more common perception modeling. However, the reviewer is not knowledgeable in using ODEs for deep networks. Contributions of the paper : The contribution of the paper is in proposing a learnable activation function in form of an ODE which can help to better model highly oscillatory and irregular functions more efficiently. This can be potentially useful for special applications in inverse problems where (by field knowledge) we know highly non-linear and specific activation functions can be more reasonable than the common ReLU and its recent variants. Quality and composition: The composition of the theoretical part of the paper is clear. The experimental part is very limited though and does not include all the necessary details. Many of the details in the initial sections can be taken to the appendices to make room for more empirical studies. Novelty, related works: The work seems novel in proposing the specific activation function but in general there are many other works that propose learnable or more elaborate activation functions, neurons, or local parts of networks. It is not so clear how this work is different from those and nor is compared to those works. This includes networks in networks, maxout networks, capsule networks, etc. Critique of the theories and experiments: Theoretical Design: The theoretical design and derivation of the paper seem correct, although the reviewer is not an expert on this topic. However, it does not clearly mention why the ODE is not designed and solved for each problem separately. Should there be different design choices for y for each task/dataset? Why not solving the ODE during the training as well? If we are solving the ODE only once and based on some initialization of coefficients, it seems to be equivalent to designing a learnable activation function such as leaky ReLU. In that regard, one could call leaky-ReLU a DifEN? Experimental Setup: The motivation of the new activation function is for specific use-cases where oscillatory or decaying functions are to be modelled. In that respect, the experimental setup is quite limited and inconclusive. - MNIST experiment: to conclusively evaluate the performance of the proposed activation function, it is important to try fixed activation functions on the same architecture as the DifEN and vice versa. - Diabetes regression experiment: since the task is not a well-studied regression task, more experiments on various datasets are required to make a conclusion. - The learnable activation function can potentially make the network more prone to overfitting, this needs to be tested thoroughly. - An important application of the proposed activation function is mentioned to be for model compression. That should be properly tested on problems with large sets of parameters (such as ImageNet networks) and observe if the performance drop due to a decrease in the number of parameters in a standard network is sharper than that of a network with DifEN activations. - More tasks and more analysis should be performed for the real-world tasks that the authors mention as the motivation of this work (specifically medical diagnosis or predictions). The analysis should demonstrate and give insight on the extra generalization power that the new activation function brings to those problems. - Following on the previous point, it is important to empirically demonstrate for which applications DifEN is useful. - Some analyses are missing on when a neuron can accentuate the problem of vanishing and/or exploding gradients in certain configurations of the DifEN parameters. It seems like the situation can arise during the training where the activation function become too steep or too saturated. Summary judgment: All in all, I think the proposed work has some potential in specific applications, however, the experimental setup does not give a clear and conclusive message of where and how the new activation functions are useful.
assistant
 Personal Expertise: The reviewer has extensive practical and theoretical experience with deep networks, different activation functions, as well as some practical experience using deep networks to model physical phenomena which are different inverse problems than the more common perception modeling. However, the reviewer is not knowledgeable in using ODEs for deep networks. Contributions of the paper : The contribution of the paper is in proposing a learnable activation function in form of an ODE which can help to better model highly oscillatory and irregular functions more efficiently. This can be potentially useful for special applications in inverse problems where (by field knowledge) we know highly non-linear and specific activation functions can be more reasonable than the common ReLU and its recent variants. Quality and composition: The composition of the theoretical part of the paper is clear. The experimental part is very limited though and does not include all the necessary details. Many of the details in the initial sections can be taken to the appendices to make room for more empirical studies. Novelty, related works: The work seems novel in proposing the specific activation function but in general there are many other works that propose learnable or more elaborate activation functions, neurons, or local parts of networks. It is not so clear how this work is different from those and nor is compared to those works. This includes networks in networks, maxout networks, capsule networks, etc. Critique of the theories and experiments: Theoretical Design: The theoretical design and derivation of the paper seem correct, although the reviewer is not an expert on this topic. However, it does not clearly mention why the ODE is not designed and solved for each problem separately. Should there be different design choices for y for each task/dataset? Why not solving the ODE during the training as well? If we are solving the ODE only once and based on some initialization of coefficients, it seems to be equivalent to designing a learnable activation function such as leaky ReLU. In that regard, one could call leaky-ReLU a DifEN? Experimental Setup: The motivation of the new activation function is for specific use-cases where oscillatory or decaying functions are to be modelled. In that respect, the experimental setup is quite limited and inconclusive. - MNIST experiment: to conclusively evaluate the performance of the proposed activation function, it is important to try fixed activation functions on the same architecture as the DifEN and vice versa. - Diabetes regression experiment: since the task is not a well-studied regression task, more experiments on various datasets are required to make a conclusion. - The learnable activation function can potentially make the network more prone to overfitting, this needs to be tested thoroughly. - An important application of the proposed activation function is mentioned to be for model compression. That should be properly tested on problems with large sets of parameters (such as ImageNet networks) and observe if the performance drop due to a decrease in the number of parameters in a standard network is sharper than that of a network with DifEN activations. - More tasks and more analysis should be performed for the real-world tasks that the authors mention as the motivation of this work (specifically medical diagnosis or predictions). The analysis should demonstrate and give insight on the extra generalization power that the new activation function brings to those problems. - Following on the previous point, it is important to empirically demonstrate for which applications DifEN is useful. - Some analyses are missing on when a neuron can accentuate the problem of vanishing and/or exploding gradients in certain configurations of the DifEN parameters. It seems like the situation can arise during the training where the activation function become too steep or too saturated. Summary judgment: All in all, I think the proposed work has some potential in specific applications, however, the experimental setup does not give a clear and conclusive message of where and how the new activation functions are useful.