Answers.Body: <p>Yes, there is such a statement, valid even in a bit more general setting.</p>
  <p>Any function, equivariant to a certain symmetry, can be approximated arbitrarily well, provided that the number of parameters is sufficient.</p>
  <p>For the case of CNN, the symmetry is translational, and with a sufficient number of filters, any translationally equivariant function can be approximated by a suitable CNN.</p>
  <p>For the details and exposition, please, look at the paper <a href="https://arxiv.org/abs/1804.10306" rel="nofollow noreferrer">Universal approximations of invariant maps by neural networks</a> (2018) by Dmitry Yarotsky.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-02T12:35:05.993Z
Answers.Text: Hello. I just want to let you know that, whenever you're providing a reference, I think it is a good idea to provide not just the link but also its name/title, because the link can become broken.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-03T15:49:57.010Z
Answers.Text: @nbro sorry for that, I'll keep in mind)
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-31T20:16:42.897Z
Answers.LastActivityDate: 2021-11-02T12:33:34.743Z
Answers.LastEditDate: 2021-11-02T12:33:34.743Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about &quot;single-channel&quot; standard networks (multi-layer perceptrons), where all layers are 2D vectors and the weights can be represented in weight matrices.</p>
  <p>In particular, this is no longer applicable in some convolutional network applications, where the layers tend to be tensors with multiple feature channels. Of course, one could construct an equivalent &quot;single-channel&quot; neural network from a multi-channel network by putting the weight matrices together in a certain way. However, one would then have sparse matrices as weight matrices with very many constraints on the matrix entries, so that the &quot;standard theorems&quot; would no longer be applicable.</p>
  <p><strong>Do you know of any papers that study the Universal Approximation Theorem for multi-channel neural networks? Or is there a way to derive it from one of the other theorems?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-02T22:51:27.483Z
Comments.Text: For "multi-channel neural networks" do you mean (just) _convolutional neural networks_? If not, can you please clarify what other "multi-channel" neural networks are you referring to?
Comments.UserDisplayName: nbro
Tags: <neural-networks><convolutional-neural-networks><reference-request><universal-approximation-theorems>
Title: Is there any paper that shows that multi-channel neural networks are universal approximators?
UserDisplayName: Niclas
ThreadText: # Is there any paper that shows that multi-channel neural networks are universal approximators?
  
  (Asked by: username_0 on 2021-10-31)
  
  Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where all layers are 2D vectors and the weights can be represented in weight matrices.
  
  In particular, this is no longer applicable in some convolutional network applications, where the layers tend to be tensors with multiple feature channels. Of course, one could construct an equivalent "single-channel" neural network from a multi-channel network by putting the weight matrices together in a certain way. However, one would then have sparse matrices as weight matrices with very many constraints on the matrix entries, so that the "standard theorems" would no longer be applicable.
  
  **Do you know of any papers that study the Universal Approximation Theorem for multi-channel neural networks? Or is there a way to derive it from one of the other theorems?**
  
  **Tags:** neural-networks, convolutional-neural-networks, reference-request, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: For "multi-channel neural networks" do you mean (just) _convolutional neural networks_? If not, can you please clarify what other "multi-channel" neural networks are you referring to?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-10-31. Score: 1 | Accepted answer
  
  Yes, there is such a statement, valid even in a bit more general setting.
  
  Any function, equivariant to a certain symmetry, can be approximated arbitrarily well, provided that the number of parameters is sufficient.
  
  For the case of CNN, the symmetry is translational, and with a sufficient number of filters, any translationally equivariant function can be approximated by a suitable CNN.
  
  For the details and exposition, please, look at the paper [Universal approximations of invariant maps by neural networks](https://arxiv.org/abs/1804.10306) (2018) by Dmitry Yarotsky.
  
  > **Comments:**
  > **username_1**: Hello. I just want to let you know that, whenever you're providing a reference, I think it is a good idea to provide not just the link but also its name/title, because the link can become broken.  
  > **username_2**: @username_1 sorry for that, I'll keep in mind)  
  
  ---
  
  