Answers.Body: <p>In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that <span class="math-container">$\rho$</span> is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter <span class="math-container">$\alpha$</span>.</p>
  
  <p>In this paper, <span class="math-container">$\rho$</span> is a non-linear function (that is, a function that is not <a href="https://en.wikipedia.org/wiki/Linear_function" rel="nofollow noreferrer">linear</a>, e.g. the <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">sigmoid function</a>) that applies element-wise to its input. So, if you pass a vector to this <span class="math-container">$\rho$</span>, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but <span class="math-container">$\rho$</span> does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, <span class="math-container">$\rho$</span> is also followed by a matrix. In general, this is not forbidden, even though it is not common. </p>
  
  <p>In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-21T20:07:25.053Z
Answers.LastActivityDate: 2019-05-21T20:07:25.053Z
Answers.UserDisplayName: nbro
Body: <p>In the paper <a href="https://arxiv.org/pdf/1710.06234.pdf" rel="nofollow noreferrer">Nonlinear Interference Mitigation via Deep Neural Networks</a>, the the following network is illustrated.</p>
  
  <p>The network structure is <a href="https://i.sstatic.net/1rsp2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1rsp2.png" alt="neural network structure"></a></p>
  
  <p>The network parameters are <span class="math-container">$\theta = \{W_1^{1},...,W_1^{l-1},W_2^{1},...,W_2^{l-1},W^{l},\alpha_1,...,\alpha_{l-1}\}$</span>, where <span class="math-container">$W_1$</span> and <span class="math-container">$W_2$</span> are linear matrices and <span class="math-container">$\rho^{(i)}(x)=xe^{-j\alpha_i|x|^2}$</span> is element-wise nonlinear function (<span class="math-container">$i$</span> is the index of layer). </p>
  
  <p>Where should I add this <span class="math-container">$\rho^{(i)}(x)$</span>? Is it possible to learn the parameter <span class="math-container">$\alpha$</span>? I don't think it is the same idea as activation function since it is positioned in the middle of two linear matrices... Or can it be added as embedding layer?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T13:37:48.770Z
Comments.Text: Hi and welcome to this community! Can you please tell us which paper you found these expressions in?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T13:42:27.307Z
Comments.Text: Hi thanks! It's in the field of optical communication. check https://arxiv.org/pdf/1710.06234.pdf Figure 1 bottom branch
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:01:59.677Z
Comments.Text: After a quick look at the relevant parts of the paper, I think that $\rho$ are the activation functions that act element-wise (so if the input to $\rho$ is a vector, the result is also a vector with the same shape as the input vector). However, the activation functions are also followed by another linear transformation, the matrices $W_2^{l}$, where $l$ is the layer. $\alpha$ should be learnable parameters (complex numbers), according to the paper.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:05:22.963Z
Comments.Text: Thank you very much, to my understanding, at the end of each layer there should be activation function, so how do you add another linear transformation after that? And is there a way to learn parameters in the activation function?
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:06:58.980Z
Comments.Text: It is usual the case that the non-linearity is applied after the linear transformations, but, in this architecture, it seems not to be the case: I have not read more than a few lines of the paper, so maybe they explain why they are doing this. They say that the non-linearity is a differentiable function. So, you can take the derivative of this function w.r.t. to its parameter $\alpha$. Once you have the derivative of a differentiable function w.r.t. to one of its parameters, you can use gradient descent to learn those parameters.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:10:48.510Z
Comments.Text: Thank you again. And is it possible to learn $W_1$ and $W_2$ at the same time? From what I learned there is only one weight matrix for each layer, so two weight matrices still make sense then? Learning parameters can be $\alpha$ in addition to $W$ and $b$?
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:13:21.457Z
Comments.Text: You can have more than one matrix associated with a single layer. For example, in a recurrent neural network, for each layer, you have the weight matrix associated with the feed-forward connections and you have another weight matrix associated with the recurrent connections. So, in general, you can have more than one weight matrix for each layer. The way you combine these matrices with the inputs defines also the architecture. So, yes, you can learn both $W_1$ and $W_2$ at the same time (that is, you can update them at the same gradient descent step).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:14:21.277Z
Comments.Text: Yes, in general, you can have different parameters and not just the weight matrices. You just need to be able to calculate the partial derivatives of the loss function w.r.t. those parameters.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:28:35.193Z
Comments.Text: Thank you so much for your answer! So basically I have to construct my own network functions and not use keras to add dense layer and activation function that way right?
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:33:12.650Z
Comments.Text: Yes, you may need to do that, that is, create a custom layer with your specific learnable parameters (but I am not super familiar with Keras).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:34:44.587Z
Comments.Text: Thank you so much for your help! I should learn more and come back later!! Many thanks!
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:36:48.090Z
Comments.Text: One more question, could you please recommend some study material for this specific question? I know it's a lot to ask and I really appreciate your help!
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:42:47.533Z
Comments.Text: The tensorflow tutorials are mostly based on high-level API and there's not much room for customization.
Comments.UserDisplayName: yfliuuu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T14:55:15.620Z
Comments.Text: You should be able to implement this model in TensorFlow (but I am not sure), given it's one of the more complete ML frameworks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-21T17:36:45.840Z
Comments.Text: Really appreciate it!
Comments.UserDisplayName: yfliuuu
Tags: <deep-learning><deep-neural-networks>
Title: Can you learn parameters in nonlinear function?
UserDisplayName: yfliuuu
ThreadText: # Can you learn parameters in nonlinear function?
  
  (Asked by: username_0 on 2019-05-21)
  
  In the paper [Nonlinear Interference Mitigation via Deep Neural Networks](https://arxiv.org/pdf/1710.06234.pdf), the the following network is illustrated.
  
  The network structure is [![neural network structure](https://i.sstatic.net/1rsp2.png)](https://i.sstatic.net/1rsp2.png)
  
  The network parameters are $\theta = \{W\_1^{1},...,W\_1^{l-1},W\_2^{1},...,W\_2^{l-1},W^{l},\alpha\_1,...,\alpha\_{l-1}\}$, where $W\_1$ and $W\_2$ are linear matrices and $\rho^{(i)}(x)=xe^{-j\alpha\_i|x|^2}$ is element-wise nonlinear function ($i$ is the index of layer).
  
  Where should I add this $\rho^{(i)}(x)$? Is it possible to learn the parameter $\alpha$? I don't think it is the same idea as activation function since it is positioned in the middle of two linear matrices... Or can it be added as embedding layer?
  
  **Tags:** deep-learning, deep-neural-networks
  > **Comments:**
  > **username_1**: Hi and welcome to this community! Can you please tell us which paper you found these expressions in?  
  > **username_0**: Hi thanks! It's in the field of optical communication. check https://arxiv.org/pdf/1710.06234.pdf Figure 1 bottom branch  
  > **username_1**: After a quick look at the relevant parts of the paper, I think that $\rho$ are the activation functions that act element-wise (so if the input to $\rho$ is a vector, the result is also a vector with the same shape as the input vector). However, the activation functions are also followed by another linear transformation, the matrices $W_2^{l}$, where $l$ is the layer. $\alpha$ should be learnable parameters (complex numbers), according to the paper.  
  > **username_0**: Thank you very much, to my understanding, at the end of each layer there should be activation function, so how do you add another linear transformation after that? And is there a way to learn parameters in the activation function?  
  > **username_1**: It is usual the case that the non-linearity is applied after the linear transformations, but, in this architecture, it seems not to be the case: I have not read more than a few lines of the paper, so maybe they explain why they are doing this. They say that the non-linearity is a differentiable function. So, you can take the derivative of this function w.r.t. to its parameter $\alpha$. Once you have the derivative of a differentiable function w.r.t. to one of its parameters, you can use gradient descent to learn those parameters.  
  > **username_0**: Thank you again. And is it possible to learn $W_1$ and $W_2$ at the same time? From what I learned there is only one weight matrix for each layer, so two weight matrices still make sense then? Learning parameters can be $\alpha$ in addition to $W$ and $b$?  
  > **username_1**: You can have more than one matrix associated with a single layer. For example, in a recurrent neural network, for each layer, you have the weight matrix associated with the feed-forward connections and you have another weight matrix associated with the recurrent connections. So, in general, you can have more than one weight matrix for each layer. The way you combine these matrices with the inputs defines also the architecture. So, yes, you can learn both $W_1$ and $W_2$ at the same time (that is, you can update them at the same gradient descent step).  
  > **username_1**: Yes, in general, you can have different parameters and not just the weight matrices. You just need to be able to calculate the partial derivatives of the loss function w.r.t. those parameters.  
  > **username_0**: Thank you so much for your answer! So basically I have to construct my own network functions and not use keras to add dense layer and activation function that way right?  
  > **username_1**: Yes, you may need to do that, that is, create a custom layer with your specific learnable parameters (but I am not super familiar with Keras).  
  > **username_0**: Thank you so much for your help! I should learn more and come back later!! Many thanks!  
  > **username_0**: One more question, could you please recommend some study material for this specific question? I know it's a lot to ask and I really appreciate your help!  
  > **username_0**: The tensorflow tutorials are mostly based on high-level API and there's not much room for customization.  
  > **username_1**: You should be able to implement this model in TensorFlow (but I am not sure), given it's one of the more complete ML frameworks.  
  > **username_0**: Really appreciate it!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-21. Score: 1
  
  In general, you can learn any parameter of the network, provided you can find the partial derivative of the loss function with respect to the desired parameter. Given that $\rho$ is assumed to be differentiable (as the authors state in the paper), you can take the partial derivative of the loss function with respect to the parameter $\alpha$.
  
  In this paper, $\rho$ is a non-linear function (that is, a function that is not [linear](https://en.wikipedia.org/wiki/Linear_function), e.g. the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function)) that applies element-wise to its input. So, if you pass a vector to this $\rho$, you will get a vector of the same shape out of it. The authors do not explicitly call it an "activation function", but $\rho$ does an analogous job of an activation function, that is, it introduces non-linearity. Furthermore, in this architecture, $\rho$ is also followed by a matrix. In general, this is not forbidden, even though it is not common.
  
  In general, in each layer of a neural network, you can have several different learnable parameters or weights. A parameter is learnable if you can differentiate the loss function with respect to it. You can have more than one weight matrix. For example, recurrent neural networks have usually more than one weight matrix associated with each layer: one matrix is associated with the feed-forward connections and the other matrix is associated with the recurrent connections.
  
  
  ---
  
  