Activation in deep neural networks is fundamental to achieving non-linear mappings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The research on flexible activation is quite limited in both designing philosophy and application scenarios. In this study, we propose a general combined form of flexible activation functions as well as three principles of choosing flexible activation component. Based on this, we develop two novel flexible activation functions that can be implemented in LSTM cells and auto-encoder layers. Also two new regularisation terms based on assumptions as prior knowledge are proposed. We find that LSTM and fully connected auto-encoder models with proposed flexible activations provides significant improvements on time series forecasting and image compressing tasks, while layer-wise regularization can improve the performance of CNN (LeNet-5) models with PReLu activation in image classification tasks.
This paper presents a family of parameterized composite activation functions, and a regularization technique for parameterized activation functions in general. While the three sets of experiments show potentially promising results, they aren't able to disambiguate clearly between the effect of the activation function you introduce and the effect of additional parameters in general, or between the effect of the regularization technique you introduce and the effect of regularization in general. I would really like to see the kind of carefully baselined, ablated, and hyperparameter-tuned results that would justify adding the techniques introduced to the toolbox of a typical deep learning practitioner.
Some feedback:
Typos:
- The formatting is off a bit (shifted horizontally?)
- In the abstract: RPeLu -> PReLU
- p. 5: "basement settings"->baseline, "logarithmic sale"->scale
- p. 6: basement->baseline again
- p. 7 etc.: trail->trial
Feedback:
- Intro:
  - Explain why maxout is similar to training piecewise activation functions?
  - It's unclear what "making the most of the non-linear properties by introducing adaptation policy on the input" means
  - An "autoencoder" is not an architecture as much as a broad family of architectures coupled with training approaches (I'm guessing you mean a fully-connected autoencoder)
- Methodology:
  - consider using "f" instead of "a" so it's easier to tell apart from alpha?
- Experiments:
  - I'm not sure I'm convinced by the statistical tests used on the LSTM results; they demonstrate that your approach, with a few specific sets of hyperparameter settings, does better than the baseline, but not that this represents a valid claim about your activation function's effect on this LSTM model in general.
  - The autoencoder experiments are even less convincing, in that they represent two seemingly arbitrary network architectures, with equally arbitrary placement of the activation function in one of them.
  - The regularization experiments use LeNet-5, which is not a compelling benchmark architecture with respect to contemporary practice. The effects of regularization techniques can be very different in different regimes of dataset and network size.
- Code:
  - I'm not sure why you performed backprop by hand for your activation functions and used torch.Function, rather than writing them directly as nn.Modules that make use of PyTorch autograd
  - There are lots of details in the code that aren't in the paper; in general, papers should aim to be relatively self-contained (although I'm very glad to see your code, and it's pretty simple to follow)