Answers.Body: <p>Others already mentioned:</p>
  
  <ul>
  <li>zero padding</li>
  <li>RNN</li>
  <li>recursive NN</li>
  </ul>
  
  <p>so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an <a href="http://www.deeplearningbook.org/contents/convnets.html" rel="noreferrer">excellent book</a> which backs up this approach:</p>
  
  <blockquote>
    <p>Consider a collection of images, where each image has a different
    width and height. It is unclear how to model such inputs with a weight
    matrix of fixed size. Convolution is straightforward to apply; the
    kernel is simply applied a different number of times depending on the
    size of the input, and the output of the convolution operation scales
    accordingly.</p>
  </blockquote>
  
  <p>Taken from page 354, 9.7 Data Types, 3rd paragraph. You can read it further to see some other approaches.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-21T20:39:51.517Z
Answers.Text: I can't find the text you cite in your answer in the book.  Maybe it was present originally and has been removed later?  Page 354 currently has a paragraph which starts similarly, but never uses variable number of layers (but variable output size instead).
Answers.UserDisplayName: jochen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T08:15:57.997Z
Answers.Text: @Salvador Dali can "0" padding with mask be a good solution for MLPs?
Answers.UserDisplayName: DINA TAKLIT
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T21:03:32.323Z
Answers.Text: FWIW, many (most?) implementations of modern CNNs for vision domains end with an adaptive pooling layer (e.g., adaptive averaging). So even though they apply convolutions a fixed number of times, and although they usually have a minimum input size, they usually don't have any maximum input size, nor do they usually have any further restrictions on aspect ratio, etc. I can imagine doing something similar for 1D varied-length data. I haven't read any papers on adaptive pooling; I just know that it's done a lot in practice.
Answers.UserDisplayName: Alexander Guyer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-01T11:34:38.530Z
Answers.Text: It occurs to me that this approach will only work if 1) the aspect ratio (AR) of all the input images is the same, 2) you rescale all images to a given AR, or 3) you zero-pad images to force a given AR.
Answers.UserDisplayName: Matti Wens
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-09-26T06:34:19.113Z
Answers.LastActivityDate: 2019-12-05T18:14:00.180Z
Answers.LastEditDate: 2019-12-05T18:14:00.180Z
Answers.UserDisplayName: Salvador Dali
Answers.Body: <p>In NLP you have an inherent ordering of the inputs so RNNs are a natural choice.</p>
  
  <p>For variable sized inputs where there is no particular ordering among the inputs, one can design networks which:</p>
  
  <ol>
  <li>use a repetition of the same subnetwork for each of the groups of inputs (i.e. with shared weights). This repeated subnetwork learns a representation of the (groups of) inputs.</li>
  <li>use an operation on the representation of the inputs which has the same symmetry as the inputs. For order invariant data, averaging the representations from the input networks is a possible choice.</li>
  <li>use an output network to minimize the loss function at the output based on the combination of the representations of the input.</li>
  </ol>
  
  <p>The structure looks as follows:</p>
  
  <p><a href="https://i.sstatic.net/x4cCi.png" rel="noreferrer"><img src="https://i.sstatic.net/x4cCi.png" alt="network structure"></a></p>
  
  <p>Similar networks have been used to learn the relations between objects (<a href="https://arxiv.org/abs/1702.05068" rel="noreferrer">arxiv:1702.05068</a>).</p>
  
  <p>A simple example of how to learning the sample variance of a variable sized set of values is given <a href="https://medium.com/@andre.holzner/learning-a-function-with-a-variable-number-of-inputs-with-pytorch-c487e35d4dba" rel="noreferrer">here</a> (disclaimer: I'm the author of the linked article).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-22T06:58:09.530Z
Answers.LastActivityDate: 2018-09-22T06:58:09.530Z
Answers.UserDisplayName: Andre Holzner
Answers.Body: <p>one other big thing which gives this ability of dealing varying input sizes, which is the most important reason that they do such, is their <code>independence of model of sequence input and out sizes</code>. so there should be no layer which one of its input or out sizes is input or out sizes of your sequence, defined. like in rnn family of neural networks, which are used for varying size sequences, as they rely on dimension of each sequence item. also we can see it in transformers which are also used for varying size sequences we can this independence, as its been demonstrated in <a href="https://github.com/FarhangAmaji/versatileAnnModule/blob/master/models/multivariateTransformers.py" rel="nofollow noreferrer">this repo of multivariate transformers for forecasting</a>(note in this models usually its only dependent on model max input size which is needed in some embeddings, but note the network itself is independent of sequence input and output sizes).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-20T21:34:08.033Z
Answers.LastActivityDate: 2023-08-20T21:34:08.033Z
Answers.UserDisplayName: Farhang Amaji
Answers.Body: <p>Three possibilities come to mind.</p>
  
  <p>The easiest is the <strong>zero-padding</strong>. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" rel="noreferrer">Recurrent NNs</a> (RNN) are a very natural NN to choose if you have texts of varying size as input. You input words as <strong>word vectors</strong> (or embeddings) just one after another and the internal state of the RNN is supposed to encode the meaning of the full string of words. <a href="http://www.iro.umontreal.ca/~lisa/pointeurs/RNNSpokenLanguage2013.pdf" rel="noreferrer">This is one</a> of the earlier papers.</p>
  
  <p>Another possibility is using <a href="https://en.wikipedia.org/wiki/Recursive_neural_network" rel="noreferrer"><strong>recursive NNs</strong></a>. This is basically a form of preprocessing in which a text is recursively reduced to a smaller number of word vectors until only one is left - your input, which is supposed to encode the whole text. This makes a lot of sense from a linguistic point of view if your input consists of sentences (which can vary a lot in size), because sentences are structured recursively. For example, the word vector for "the man", should be similar to the word vector for "the man who mistook his wife for a hat", because noun phrases act like nouns, etc. Often, you can use linguistic information to guide your recursion on the sentence. If you want to go way beyond the Wikipedia article, <a href="http://nlp.stanford.edu/~socherr/thesis.pdf" rel="noreferrer">this is probably a good start</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-25T08:13:46.690Z
Answers.Text: @BlindKungFuMaster 0 padding with mask for MLPs is it good and does not affect badly the accuracy ?
Answers.UserDisplayName: DINA TAKLIT
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-24T02:13:36.910Z
Answers.Text: What is the difference between what you called "recursive NNs", and RNNs? R is for recursive... Also, a few citations and pointers would be useful.
Answers.UserDisplayName: Eric Platon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-25T09:29:17.547Z
Answers.Text: R is for recurrent. Recurrent means adding input linearly to the same NN again and again. Recursive means inputting the data along a tree structure.
Answers.UserDisplayName: BlindKungFuMaster
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-09-23T11:27:44.700Z
Answers.LastActivityDate: 2019-07-13T19:31:32.923Z
Answers.LastEditDate: 2019-07-13T19:31:32.923Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p>As far as I can tell, neural networks have a <strong>fixed number of neurons</strong> in the input layer.</p>
  
  <p>If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a network. 
  How is the <strong>varying input size</strong> reconciled with the <strong>fixed size</strong> of the input layer of the network? In other words, how is such a network made flexible enough to deal with an input that might be anywhere from one word to multiple pages of text?</p>
  
  <p>If my assumption of a fixed number of input neurons is wrong and new input neurons are added to/removed from the network to match the input size I don't see how these can ever be trained.</p>
  
  <p>I give the example of NLP, but lots of problems have an inherently unpredictable input size. I'm interested in the general approach for dealing with this.</p>
  
  <p>For images, it's clear you can up/downsample to a fixed size, but, for text, this seems to be an impossible approach since adding/removing text changes the meaning of the original input.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-28T00:51:50.433Z
Comments.Text: can you clarify what you mean by downsampling to a fixed size? How is downsampling done?
Comments.UserDisplayName: Charlie Parker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-28T01:02:03.940Z
Comments.Text: related: https://stats.stackexchange.com/questions/388859/is-it-possible-to-give-variable-sized-images-as-input-to-convolutioal-neural-net
Comments.UserDisplayName: Charlie Parker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-17T06:25:22.913Z
Comments.Text: Whereas the original question is fairly open-ended, the answers focus primarily on NLP. However, I stumbled on this question while looking how to do variable size image inputs for a CNN. Variable size inputs are indeed possible for a convolutional approach - albeit with some caveats, and the stats.stackexchange link above explores that related, alternate line of inquiry.
Comments.UserDisplayName: J Trana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-09T07:44:01.610Z
Comments.Text: A similar question: [How to input Scikit learn MLP classifier with variable length of input data.](https://stackoverflow.com/questions/50216691/how-to-input-scikit-learn-mlp-classifier-with-variable-length-of-input-data/70286197#70286197)
Comments.UserDisplayName: Fang WU
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T19:29:34.817Z
Comments.Text: I'm curious why no one mentions **Fully Convolutional Networks** and **Spatial Pyramid Pooling**.
Comments.UserDisplayName: huang
Tags: <neural-networks><machine-learning><deep-learning><convolutional-neural-networks><natural-language-processing>
Title: How can neural networks deal with varying input sizes?
UserDisplayName: Asciiom
ThreadText: # How can neural networks deal with varying input sizes?
  
  (Asked by: username_0 on 2016-09-23)
  
  As far as I can tell, neural networks have a **fixed number of neurons** in the input layer.
  
  If neural networks are used in a context like NLP, sentences or blocks of text of varying sizes are fed to a network.
  How is the **varying input size** reconciled with the **fixed size** of the input layer of the network? In other words, how is such a network made flexible enough to deal with an input that might be anywhere from one word to multiple pages of text?
  
  If my assumption of a fixed number of input neurons is wrong and new input neurons are added to/removed from the network to match the input size I don't see how these can ever be trained.
  
  I give the example of NLP, but lots of problems have an inherently unpredictable input size. I'm interested in the general approach for dealing with this.
  
  For images, it's clear you can up/downsample to a fixed size, but, for text, this seems to be an impossible approach since adding/removing text changes the meaning of the original input.
  
  **Tags:** neural-networks, machine-learning, deep-learning, convolutional-neural-networks, natural-language-processing
  > **Comments:**
  > **username_1**: can you clarify what you mean by downsampling to a fixed size? How is downsampling done?  
  > **username_1**: related: https://stats.stackexchange.com/questions/388859/is-it-possible-to-give-variable-sized-images-as-input-to-convolutioal-neural-net  
  > **username_2**: Whereas the original question is fairly open-ended, the answers focus primarily on NLP. However, I stumbled on this question while looking how to do variable size image inputs for a CNN. Variable size inputs are indeed possible for a convolutional approach - albeit with some caveats, and the stats.stackexchange link above explores that related, alternate line of inquiry.  
  > **username_3**: A similar question: [How to input Scikit learn MLP classifier with variable length of input data.](https://stackoverflow.com/questions/50216691/how-to-input-scikit-learn-mlp-classifier-with-variable-length-of-input-data/70286197#70286197)  
  > **username_4**: I'm curious why no one mentions **Fully Convolutional Networks** and **Spatial Pyramid Pooling**.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_12 on 2016-09-23. Score: 73 | Accepted answer
  
  Three possibilities come to mind.
  
  The easiest is the **zero-padding**. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty limited and certainly not useful if your input ranges from a few words to full texts.
  
  [Recurrent NNs](https://en.wikipedia.org/wiki/Recurrent_neural_network) (RNN) are a very natural NN to choose if you have texts of varying size as input. You input words as **word vectors** (or embeddings) just one after another and the internal state of the RNN is supposed to encode the meaning of the full string of words. [This is one](http://www.iro.umontreal.ca/~lisa/pointeurs/RNNSpokenLanguage2013.pdf) of the earlier papers.
  
  Another possibility is using [**recursive NNs**](https://en.wikipedia.org/wiki/Recursive_neural_network). This is basically a form of preprocessing in which a text is recursively reduced to a smaller number of word vectors until only one is left - your input, which is supposed to encode the whole text. This makes a lot of sense from a linguistic point of view if your input consists of sentences (which can vary a lot in size), because sentences are structured recursively. For example, the word vector for "the man", should be similar to the word vector for "the man who mistook his wife for a hat", because noun phrases act like nouns, etc. Often, you can use linguistic information to guide your recursion on the sentence. If you want to go way beyond the Wikipedia article, [this is probably a good start](http://nlp.stanford.edu/~socherr/thesis.pdf).
  
  > **Comments:**
  > **username_13**: What is the difference between what you called "recursive NNs", and RNNs? R is for recursive... Also, a few citations and pointers would be useful.  
  > **username_12**: R is for recurrent. Recurrent means adding input linearly to the same NN again and again. Recursive means inputting the data along a tree structure.  
  > **username_7**: @BlindKungFuMaster 0 padding with mask for MLPs is it good and does not affect badly the accuracy ?  
  
  ---
  
  ### Answer by username_5 on 2016-09-26. Score: 22
  
  Others already mentioned:
  
  * zero padding
  * RNN
  * recursive NN
  
  so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an [excellent book](http://www.deeplearningbook.org/contents/convnets.html) which backs up this approach:
  
  > Consider a collection of images, where each image has a different
  > width and height. It is unclear how to model such inputs with a weight
  > matrix of fixed size. Convolution is straightforward to apply; the
  > kernel is simply applied a different number of times depending on the
  > size of the input, and the output of the convolution operation scales
  > accordingly.
  
  Taken from page 354, 9.7 Data Types, 3rd paragraph. You can read it further to see some other approaches.
  
  > **Comments:**
  > **username_9**: It occurs to me that this approach will only work if 1) the aspect ratio (AR) of all the input images is the same, 2) you rescale all images to a given AR, or 3) you zero-pad images to force a given AR.  
  > **username_7**: @Salvador Dali can "0" padding with mask be a good solution for MLPs?  
  > **username_6**: I can't find the text you cite in your answer in the book.  Maybe it was present originally and has been removed later?  Page 354 currently has a paragraph which starts similarly, but never uses variable number of layers (but variable output size instead).  
  > **username_8**: FWIW, many (most?) implementations of modern CNNs for vision domains end with an adaptive pooling layer (e.g., adaptive averaging). So even though they apply convolutions a fixed number of times, and although they usually have a minimum input size, they usually don't have any maximum input size, nor do they usually have any further restrictions on aspect ratio, etc. I can imagine doing something similar for 1D varied-length data. I haven't read any papers on adaptive pooling; I just know that it's done a lot in practice.  
  
  ---
  
  ### Answer by username_10 on 2018-09-22. Score: 13
  
  In NLP you have an inherent ordering of the inputs so RNNs are a natural choice.
  
  For variable sized inputs where there is no particular ordering among the inputs, one can design networks which:
  
  1. use a repetition of the same subnetwork for each of the groups of inputs (i.e. with shared weights). This repeated subnetwork learns a representation of the (groups of) inputs.
  2. use an operation on the representation of the inputs which has the same symmetry as the inputs. For order invariant data, averaging the representations from the input networks is a possible choice.
  3. use an output network to minimize the loss function at the output based on the combination of the representations of the input.
  
  The structure looks as follows:
  
  [![network structure](https://i.sstatic.net/x4cCi.png)](https://i.sstatic.net/x4cCi.png)
  
  Similar networks have been used to learn the relations between objects ([arxiv:1702.05068](https://arxiv.org/abs/1702.05068)).
  
  A simple example of how to learning the sample variance of a variable sized set of values is given [here](https://medium.com/@andre.holzner/learning-a-function-with-a-variable-number-of-inputs-with-pytorch-c487e35d4dba) (disclaimer: I'm the author of the linked article).
  
  
  ---
  
  ### Answer by username_11 on 2023-08-20. Score: 0
  
  one other big thing which gives this ability of dealing varying input sizes, which is the most important reason that they do such, is their `independence of model of sequence input and out sizes`. so there should be no layer which one of its input or out sizes is input or out sizes of your sequence, defined. like in rnn family of neural networks, which are used for varying size sequences, as they rely on dimension of each sequence item. also we can see it in transformers which are also used for varying size sequences we can this independence, as its been demonstrated in [this repo of multivariate transformers for forecasting](https://github.com/FarhangAmaji/versatileAnnModule/blob/master/models/multivariateTransformers.py)(note in this models usually its only dependent on model max input size which is needed in some embeddings, but note the network itself is independent of sequence input and output sizes).
  
  
  ---
  
  