Answers.Body: <p>Use another network <em><strong>h</strong></em> which takes <em><strong>f(x)</strong></em> and <em><strong>g(x)</strong></em> as input i.e. <em><strong>h(f(x), g(x))</strong></em>.</p>
  <p>Training psuedo code(pytorch):</p>
  <pre><code>for epoch in epochs:
      for batch, x in dataset:
          train(f)
          train(g)
  
  
  for epoch in epochs:
      for batch, x in dataset:
          # freeze f and g (using torch.no_grad in pytorch)
          with torch.no_grad:
              fx = f(x)
              gx = g(x)
          pred = h(fx, gx)
          loss = loss_fn(pred, (fx . gx))
          backpropagate()
          optimise()
          
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T16:33:08.050Z
Answers.Text: thank you for your answer but i dont want to know how to implement it but how to write it theoretically as a neural net
Answers.UserDisplayName: Quappojoice
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T11:44:27.610Z
Answers.LastActivityDate: 2020-08-16T11:44:27.610Z
Answers.UserDisplayName: Girish Hegde
Answers.Body: <p>You need to express the product of the activation functions of your neurons via some combination of individual activation functions.</p>
  <p>For example, if the product of your activation functions is equal to the linear combination of such functions then the network for <span class="math-container">$f(x)g(x)$</span> can be derived from the networks for <span class="math-container">$f(x)$</span> and <span class="math-container">$g(x)$</span> exactly.</p>
  <p>Let's assume for simplicity that <span class="math-container">$f(x)$</span> and <span class="math-container">$g(x)$</span> are approximated with trained networks that consist of just one neuron,
  <span class="math-container">$$f(x) = A(w_f x + b_f)$$</span>
  <span class="math-container">$$g(x) = A(w_g x + b_g)$$</span>
  Then, if the activation function <span class="math-container">$A$</span> is such that
  <span class="math-container">$$A(w_f x + b_f)A(w_g x + b_g) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$</span>
  then <span class="math-container">$f(x)g(x)$</span> is approximated by a neural network consisting of <span class="math-container">$N$</span> neurons:
  <span class="math-container">$$f(x)g(x) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$</span>
  This is easily generalized to a network that consists of multiple neurons.</p>
  <p>An example is when some of your neurons have activation functions <span class="math-container">$A(z)=\cos(z)$</span> and others are <span class="math-container">$A(z)=\sin(z)$</span>. Then you can build the network computing <span class="math-container">$f(x)g(x)$</span> from the networks computing <span class="math-container">$f(x)$</span> and <span class="math-container">$g(x)$</span> using the product-to-sum identities, like <span class="math-container">$2\sin(\theta)\sin(\phi) = cos(\theta-\phi)-cos(\theta+\phi)$</span>, etc.</p>
  <p>If the product of your activation functions cannot be expressed as a linear combination of such functions (as in the case of sigmoids) then the network for <span class="math-container">$f(x)g(x)$</span> can be derived from the networks for <span class="math-container">$f(x)$</span> and <span class="math-container">$g(x)$</span> approximately. You need to figure out the parameters of the approximation
  <span class="math-container">$$A(z_f)A(z_g) \approx \sum\limits_{i=1}^N \alpha_iA(z_i)$$</span>
  Such parameters are: the number of terms, <span class="math-container">$N$</span>, the coefficients <span class="math-container">$\alpha_i$</span>, and the range <span class="math-container">$[z_{min}, z_{max}]$</span> where this approximation works with a desired accuracy.</p>
  <p>Linear combination is not the only opportunity. Say, your activation is <span class="math-container">$A(z)=a^z$</span>. Then, <span class="math-container">$$A(w_f x + b_f)A(w_g x + b_g)=a^{(w_f+w_g) x + (b_f+b_g)}=a^{w_{fg} x + b_{fg}}$$</span> so the weights and the biases of the product are exactly expressed via the weights and the biases of the multiplicands.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-23T16:56:10.623Z
Answers.LastActivityDate: 2020-08-23T17:03:06.497Z
Answers.LastEditDate: 2020-08-23T17:03:06.497Z
Answers.UserDisplayName: Vladislav Gladkikh
Body: <p>I have two functions <span class="math-container">$f(x)$</span> and <span class="math-container">$g(x)$</span>, and each of them can be computed with a neural network <span class="math-container">$\phi_f$</span> and <span class="math-container">$\phi_g$</span>.</p>
  <p>My question is, how can I write a neural net for <span class="math-container">$f(x)g(x)$</span>?</p>
  <p>So, for example, if <span class="math-container">$g(x)$</span> is constant and equal to <span class="math-container">$c$</span>  and <span class="math-container">$\phi_f = ((A_1,b_1),...(A_L,b_L))$</span>, then <span class="math-container">$\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$</span>.</p>
  <p>Actually, I need to show it for <span class="math-container">$f(x)=x$</span> and <span class="math-container">$g(x)=x^2$</span> if this make something easier.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-17T01:01:21.350Z
Comments.Text: Did you mean $\phi_{fg} = ((cA_1,cb_1),...,(cA_L,cb_L))$ rather than $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-17T08:12:40.000Z
Comments.Text: no because only the output should be multiplied with c
Comments.UserDisplayName: Quappojoice
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-17T18:37:15.043Z
Comments.Text: In the title you write $f(x) * g(x)$ and in the body $f(x)g(x)$. Is that a convolution or multiplication? If $g(x)$ is constant then you would have $(cA_L, cb_L)$ only if the activation function of the final layer is linear so it's not true in general.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-17T23:46:20.300Z
Comments.Text: @Brale Sorry, it was me who rewrote the title. It's multiplication in all cases.
Comments.UserDisplayName: nbro
Tags: <neural-networks><ai-design>
Title: Given two neural networks that compute two functions $f(x)$ and $g(x)$, how can I create a neural network that computes $f(x)g(x)$?
UserDisplayName: Quappojoice
ThreadText: # Given two neural networks that compute two functions $f(x)$ and $g(x)$, how can I create a neural network that computes $f(x)g(x)$?
  
  (Asked by: username_0 on 2020-08-16)
  
  I have two functions $f(x)$ and $g(x)$, and each of them can be computed with a neural network $\phi\_f$ and $\phi\_g$.
  
  My question is, how can I write a neural net for $f(x)g(x)$?
  
  So, for example, if $g(x)$ is constant and equal to $c$ and $\phi\_f = ((A\_1,b\_1),...(A\_L,b\_L))$, then $\phi\_{fg} = ((A\_1,b\_1),...,(cA\_L,cb\_L))$.
  
  Actually, I need to show it for $f(x)=x$ and $g(x)=x^2$ if this make something easier.
  
  **Tags:** neural-networks, ai-design
  > **Comments:**
  > **username_1**: Did you mean $\phi_{fg} = ((cA_1,cb_1),...,(cA_L,cb_L))$ rather than $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$?  
  > **username_0**: no because only the output should be multiplied with c  
  > **username_2**: In the title you write $f(x) * g(x)$ and in the body $f(x)g(x)$. Is that a convolution or multiplication? If $g(x)$ is constant then you would have $(cA_L, cb_L)$ only if the activation function of the final layer is linear so it's not true in general.  
  > **username_1**: @Brale Sorry, it was me who rewrote the title. It's multiplication in all cases.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-08-16. Score: 1
  
  Use another network h which takes f(x) and g(x) as input i.e. h(f(x), g(x)).
  Training psuedo code(pytorch):
  for epoch in epochs:
      for batch, x in dataset:
          train(f)
          train(g)
  
  
  for epoch in epochs:
      for batch, x in dataset:
          # freeze f and g (using torch.no_grad in pytorch)
          with torch.no_grad:
              fx = f(x)
              gx = g(x)
          pred = h(fx, gx)
          loss = loss_fn(pred, (fx . gx))
          backpropagate()
          optimise()
  
  > **Comments:**
  > **username_0**: thank you for your answer but i dont want to know how to implement it but how to write it theoretically as a neural net  
  
  ---
  
  ### Answer by username_4 on 2020-08-23. Score: 1
  
  You need to express the product of the activation functions of your neurons via some combination of individual activation functions.
  
  For example, if the product of your activation functions is equal to the linear combination of such functions then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ exactly.
  
  Let's assume for simplicity that $f(x)$ and $g(x)$ are approximated with trained networks that consist of just one neuron,
  $$f(x) = A(w\_f x + b\_f)$$
  $$g(x) = A(w\_g x + b\_g)$$
  Then, if the activation function $A$ is such that
  $$A(w\_f x + b\_f)A(w\_g x + b\_g) = \sum\limits\_{i=1}^N \alpha\_iA(w\_i x + b\_i)$$
  then $f(x)g(x)$ is approximated by a neural network consisting of $N$ neurons:
  $$f(x)g(x) = \sum\limits\_{i=1}^N \alpha\_iA(w\_i x + b\_i)$$
  This is easily generalized to a network that consists of multiple neurons.
  
  An example is when some of your neurons have activation functions $A(z)=\cos(z)$ and others are $A(z)=\sin(z)$. Then you can build the network computing $f(x)g(x)$ from the networks computing $f(x)$ and $g(x)$ using the product-to-sum identities, like $2\sin(\theta)\sin(\phi) = cos(\theta-\phi)-cos(\theta+\phi)$, etc.
  
  If the product of your activation functions cannot be expressed as a linear combination of such functions (as in the case of sigmoids) then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ approximately. You need to figure out the parameters of the approximation
  $$A(z\_f)A(z\_g) \approx \sum\limits\_{i=1}^N \alpha\_iA(z\_i)$$
  Such parameters are: the number of terms, $N$, the coefficients $\alpha\_i$, and the range $[z\_{min}, z\_{max}]$ where this approximation works with a desired accuracy.
  
  Linear combination is not the only opportunity. Say, your activation is $A(z)=a^z$. Then, $$A(w\_f x + b\_f)A(w\_g x + b\_g)=a^{(w\_f+w\_g) x + (b\_f+b\_g)}=a^{w\_{fg} x + b\_{fg}}$$ so the weights and the biases of the product are exactly expressed via the weights and the biases of the multiplicands.
  
  
  ---
  
  