Answers.Body: <p>This should be possible but I've never seen it done in practice. <strong>Whether or not this will even actually work is unclear to me</strong> and will be highly dependent both on your training data and choice of loss. <strong>I'd take a step back and look into the literature to see if you can't find a more established approach to your problem</strong>, perhaps with RNNs. That being said, I believe the following should do what you're asking.</p>
  <p>Consider network <span class="math-container">$N$</span> to be a dense neural net with <span class="math-container">$k$</span> layers, <span class="math-container">$N_i$</span> to be the <span class="math-container">$i$</span>th layer of <span class="math-container">$N$</span>, <span class="math-container">$L$</span> to be the max length of the input, and <span class="math-container">$V$</span> to be the number of terms in the input (the number of <span class="math-container">$v$</span>s). To accomplish what you want in the above scenario, you can add three additional layers to <span class="math-container">$N$</span>, <span class="math-container">$N_{k+1}$</span> <span class="math-container">$N_{k+2}$</span>, and <span class="math-container">$N_{k+3}$</span>:</p>
  <p><span class="math-container">$N_{k+1}$</span> is a simple dense layer that has <span class="math-container">$L$</span> neurons and takes as input the output of <span class="math-container">$N_k$</span>. This layer can be skipped if layer <span class="math-container">$N_k$</span> already has <span class="math-container">$L$</span> neurons.</p>
  <p><span class="math-container">$N_{k+2}$</span> takes as input the output <span class="math-container">$N_{k+1}$</span> and takes the Hadamard product (elementwise multiplication) of it with a second input, a binary vector of length <span class="math-container">$L$</span> with a prefix of <span class="math-container">$V$</span> <span class="math-container">$1$</span>s and suffix of <span class="math-container">$L-V$</span> <span class="math-container">$0$</span>s. For example if <span class="math-container">$L=5$</span> and <span class="math-container">$V=2$</span>, you would supply as the second input the vector <span class="math-container">$[1, 1, 0, 0, 0]$</span>, which effectively &quot;zeroes out&quot; the third, fourth, and fifth positions.</p>
  <p><span class="math-container">$N_{k+3}$</span> is your new output layer, which also has <span class="math-container">$L$</span> neurons. <span class="math-container">$N$</span> can now be trained and, given the target data is in the proper format to achieve this, it should output results like in your question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-21T11:52:45.987Z
Answers.Text: Thanks Oso for your answer. I understand the concept behind it but I would assume that putting the target data in the proper format might be quite challenging. Nontheless I will consider using your approach. Thanks for your help. I upvoted and accepted your answer.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T16:25:00.147Z
Answers.LastActivityDate: 2021-09-20T16:25:00.147Z
Answers.UserDisplayName: Andy
Body: <p>I have a general question about supervised ANNs that map inputs to outputs. It is possible to vary the length of the input and output vectors by inserting some dummy variables that will not be considered in the mapping (or will be mapped to other dummy variables). So basically the mapping should look like this (v: value, d: dummy)</p>
  <p>Input vector 1 <span class="math-container">$[v,v,v,v,v] \rightarrow$</span> Output vector 1 <span class="math-container">$[v,v,v,v,v]$</span></p>
  <p>Input vector 2 <span class="math-container">$[v,v,v,v,v]\rightarrow$</span> Output vector 2 <span class="math-container">$[v,v,v,v,v]$</span></p>
  <p>Input vector 3 <span class="math-container">$[v,v,v,d,d] \rightarrow$</span> Output vector 3 <span class="math-container">$[v,v,v,d,d]$</span></p>
  <p>Input vector 4 <span class="math-container">$[v,v,d,d,d] \rightarrow$</span>  Output vector 4 <span class="math-container">$[v,v,d,d,d]$</span></p>
  <p>Input vector 5 <span class="math-container">$[v,d,d,d,d] \rightarrow$</span>  Output vector 5 <span class="math-container">$[v,d,d,d,d]$</span></p>
  <p>The input and output vectors have a length of 5 with 5 values. However, sometimes only a vector of size e.g. 3 (which is basically a vector of length 5 with 2 dummy variables) should be mapped to an output vector of length 3. So after training the ANN should know that if it for example gets an input vector of length 3 it should produce an output vector of length 3.</p>
  <p>Is something like this generally possible with ANNs or other machine learning approaches? If so, what type of ANN or machine learning approach can be used for this? I'll appreciate every comment.</p>
  <p><strong>Reminder</strong>: Can anybody give me more insights into this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-29T10:58:47.160Z
Comments.Text: What is the difference between first two pairs of vectors?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-29T12:17:44.993Z
Comments.Text: Thanks hanugm for your answer. Basically there is no difference between the first two pair of vectors. I just wanted to show that we have multiple training data with varying length. The first two pair of vectors turn out to have the same size (same number of v) in this example.
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-29T12:18:47.123Z
Comments.Text: Im saying that first I/O vectors and second I/O vectors are same.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-29T12:22:24.570Z
Comments.Text: Well this is just an example and the v can stand for any number. The first two pair of vectors can but do not have to be identical (most probably they are not). I just have let's say 1000 input and output vectors and an ANN should be trained to map them correctly. The v stand for any number. So most of the input vectors (even if they have the same size) are not identical. But my question is trageting at having input and output vectors with different lengths
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-17T19:03:03.150Z
Comments.Text: Is the output always going to have the same length as the input? In that case you can just simply ignore the part of the output that is past the length of the input.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-17T19:04:17.793Z
Comments.Text: If the output can have variable length for a set input, then you can take a sequence to sequence approach (used in natural language processing). In that case you predict 1 embedding (word) at a time, and "end" is a possible prediction. So you can get variable length output.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-18T08:29:42.523Z
Comments.Text: Thanks Taw for your comments. Yes, the output will always have the same length as the input. I am aware that I could just ignore the part that is irrelevant. However, I assume that for the training this will not be beneficial as the ANN will learn how to map the whole sequence to the other. The ANN should just learn how the map the relevant parts of the input and output to each other. Is that somehow possible e.g. by using dummy variables that indicate to the ANN that these parts should not be considered for the mapping?
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T08:35:52.470Z
Comments.Text: @Taw: Any comments on my last comment? I'd highly appreciate every further comment from you.
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T11:53:08.993Z
Comments.Text: If you only use the relevant part of the output, the irrelevant parts will not contribute to the gradient and so they won't affect training.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T11:54:54.137Z
Comments.Text: Thanks Taw for your answer. How can I only choose the relevant part of the output (and input)? Basically the vectors have all the same length (otherwise I could not specify the ANN model).
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-23T11:30:01.010Z
Comments.Text: @Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-24T04:59:48.623Z
Comments.Text: What is the loss function? You can think about it this way; what gets sent to loss function is the concatenation of the output from NN, and the length of the input. This way the loss is computed correctly. Implementing this is just a coding question
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-24T07:44:08.977Z
Comments.Text: Thanks Taw for your answer and effort. I really appreciate it. Basically I do not understand what you meat by "what gets sent to loss function is the concatenation of the output from NN". The vectors should all have the same length. But for some of them several components should not coundt. How can I "send" this information to the loss function?
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-25T08:40:07.127Z
Comments.Text: @Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you
Comments.UserDisplayName: PeterBe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-13T10:11:16.393Z
Comments.Text: @Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you
Comments.UserDisplayName: PeterBe
Tags: <neural-networks><training>
Title: Mapping input vectors of variable length to output vectors of variable lengths with dummy variables
UserDisplayName: PeterBe
ThreadText: # Mapping input vectors of variable length to output vectors of variable lengths with dummy variables
  
  (Asked by: username_0 on 2021-07-29)
  
  I have a general question about supervised ANNs that map inputs to outputs. It is possible to vary the length of the input and output vectors by inserting some dummy variables that will not be considered in the mapping (or will be mapped to other dummy variables). So basically the mapping should look like this (v: value, d: dummy)
  
  Input vector 1 $[v,v,v,v,v] \rightarrow$ Output vector 1 $[v,v,v,v,v]$
  
  Input vector 2 $[v,v,v,v,v]\rightarrow$ Output vector 2 $[v,v,v,v,v]$
  
  Input vector 3 $[v,v,v,d,d] \rightarrow$ Output vector 3 $[v,v,v,d,d]$
  
  Input vector 4 $[v,v,d,d,d] \rightarrow$ Output vector 4 $[v,v,d,d,d]$
  
  Input vector 5 $[v,d,d,d,d] \rightarrow$ Output vector 5 $[v,d,d,d,d]$
  
  The input and output vectors have a length of 5 with 5 values. However, sometimes only a vector of size e.g. 3 (which is basically a vector of length 5 with 2 dummy variables) should be mapped to an output vector of length 3. So after training the ANN should know that if it for example gets an input vector of length 3 it should produce an output vector of length 3.
  
  Is something like this generally possible with ANNs or other machine learning approaches? If so, what type of ANN or machine learning approach can be used for this? I'll appreciate every comment.
  
  **Reminder**: Can anybody give me more insights into this?
  
  **Tags:** neural-networks, training
  
  > **Comments:**
  > **username_1**: What is the difference between first two pairs of vectors?  
  > **username_0**: Thanks username_1 for your answer. Basically there is no difference between the first two pair of vectors. I just wanted to show that we have multiple training data with varying length. The first two pair of vectors turn out to have the same size (same number of v) in this example.  
  > **username_1**: Im saying that first I/O vectors and second I/O vectors are same.  
  > **username_0**: Well this is just an example and the v can stand for any number. The first two pair of vectors can but do not have to be identical (most probably they are not). I just have let's say 1000 input and output vectors and an ANN should be trained to map them correctly. The v stand for any number. So most of the input vectors (even if they have the same size) are not identical. But my question is trageting at having input and output vectors with different lengths  
  > **username_2**: Is the output always going to have the same length as the input? In that case you can just simply ignore the part of the output that is past the length of the input.  
  > **username_2**: If the output can have variable length for a set input, then you can take a sequence to sequence approach (used in natural language processing). In that case you predict 1 embedding (word) at a time, and "end" is a possible prediction. So you can get variable length output.  
  > **username_0**: Thanks Taw for your comments. Yes, the output will always have the same length as the input. I am aware that I could just ignore the part that is irrelevant. However, I assume that for the training this will not be beneficial as the ANN will learn how to map the whole sequence to the other. The ANN should just learn how the map the relevant parts of the input and output to each other. Is that somehow possible e.g. by using dummy variables that indicate to the ANN that these parts should not be considered for the mapping?  
  > **username_0**: @username_2: Any comments on my last comment? I'd highly appreciate every further comment from you.  
  > **username_2**: If you only use the relevant part of the output, the irrelevant parts will not contribute to the gradient and so they won't affect training.  
  > **username_0**: Thanks Taw for your answer. How can I only choose the relevant part of the output (and input)? Basically the vectors have all the same length (otherwise I could not specify the ANN model).  
  > **username_0**: @username_2: Any comments to my last comment? I'd highyl appreciate every further comment from you  
  > **username_2**: What is the loss function? You can think about it this way; what gets sent to loss function is the concatenation of the output from NN, and the length of the input. This way the loss is computed correctly. Implementing this is just a coding question  
  > **username_0**: Thanks Taw for your answer and effort. I really appreciate it. Basically I do not understand what you meat by "what gets sent to loss function is the concatenation of the output from NN". The vectors should all have the same length. But for some of them several components should not coundt. How can I "send" this information to the loss function?  
  > **username_0**: @username_2: Any comments to my last comment? I'd highyl appreciate every further comment from you  
  > **username_0**: @username_2: Any comments to my last comment? I'd highyl appreciate every further comment from you  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-20. Score: 2 | Accepted answer
  
  This should be possible but I've never seen it done in practice. **Whether or not this will even actually work is unclear to me** and will be highly dependent both on your training data and choice of loss. **I'd take a step back and look into the literature to see if you can't find a more established approach to your problem**, perhaps with RNNs. That being said, I believe the following should do what you're asking.
  
  Consider network $N$ to be a dense neural net with $k$ layers, $N\_i$ to be the $i$th layer of $N$, $L$ to be the max length of the input, and $V$ to be the number of terms in the input (the number of $v$s). To accomplish what you want in the above scenario, you can add three additional layers to $N$, $N\_{k+1}$ $N\_{k+2}$, and $N\_{k+3}$:
  
  $N\_{k+1}$ is a simple dense layer that has $L$ neurons and takes as input the output of $N\_k$. This layer can be skipped if layer $N\_k$ already has $L$ neurons.
  
  $N\_{k+2}$ takes as input the output $N\_{k+1}$ and takes the Hadamard product (elementwise multiplication) of it with a second input, a binary vector of length $L$ with a prefix of $V$ $1$s and suffix of $L-V$ $0$s. For example if $L=5$ and $V=2$, you would supply as the second input the vector $[1, 1, 0, 0, 0]$, which effectively "zeroes out" the third, fourth, and fifth positions.
  
  $N\_{k+3}$ is your new output layer, which also has $L$ neurons. $N$ can now be trained and, given the target data is in the proper format to achieve this, it should output results like in your question.
  
  > **Comments:**
  > **username_0**: Thanks Oso for your answer. I understand the concept behind it but I would assume that putting the target data in the proper format might be quite challenging. Nontheless I will consider using your approach. Thanks for your help. I upvoted and accepted your answer.  
  
  ---
  
  