Answers.Body: <p>If you use RNNs, then I think the solution is to use padding (zero padding) with max sequence length (that is the max number of words in a text) in order to tell your model to skip the zeros when possible. In that way, your model will try to learn a good representation of your input with fixed size. If you do not know this dimension, a solution may be to grid search this hyperparameter.</p>
  <p>If you still want to exploit the dimensionality difference, maybe you can train different models with fixed size dimension of the representation dependently of the dimension of the input. That is, for example, use one for small, one for medium and one for large dimensions, but this should surely require to have a large and quite balanced initial dataset.</p>
  <p>Another idea could be to use the autoencoder with a fixed latent dimension. Then, do effective clustering on your samples using their latent representation, considering that similar representations should have similar dimensionality requirements (?). After that, you could train your initial dataset on k models, the same number as the clusters. That is, there should be k different latent spaces. The goal is to match each instance to the correct model. At first, you should train them all with each instance, but as the training progresses, you should maybe go with binary search for each instance in order it to find the correct model, assuming that there is total order in the measuring of the dimensionality requirements. Of course, this is just an idea, I don't know if it is going to be really helpful at all.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T19:42:50.013Z
Answers.Text: Unfortunately I can't understand your first paragraph, do you mean zero-padding the latent? can you explain more?
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T19:46:27.883Z
Answers.Text: No! In the input space in order to deal with the incosistent dimensionality.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T20:00:51.447Z
Answers.Text: Ok, but how it helps with variable-length latent vector? What I understood from this paragraph will help to have different length input (which RNNs can do with no problem at all)
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T20:00:59.943Z
Answers.Text: Your second paragraph is the way they have done in the paper I mentioned before (this one). But training a model for each latent length is not what I want, because I want the network (not me) to measure the complexity of input (useful when the input is image) and be able to produce any latent length (not those I predetermined)
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T20:31:16.993Z
Answers.Text: Check my post. I fixed it.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T17:37:37.050Z
Answers.LastActivityDate: 2020-10-18T20:30:46.737Z
Answers.LastEditDate: 2020-10-18T20:30:46.737Z
Answers.UserDisplayName: ddaedalus
Answers.Body: <p>You might want to look at an encoder-decoder sequence to sequence model. This model allows you to input and output data with variable length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T17:12:43.127Z
Answers.Text: Maybe I asked my question in a bad way. The length of input or output doesn't matter to me (and they can be both fixed). But I want the network to produce variable length latent. Like if it see a complex image, it should use more latent units to reconstruct it, but if the image of the same size is simple (e.g. mostly black), the network should decide to use less latent units and it will be able to get almost the same reconstruction error.
Answers.UserDisplayName: amin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T08:43:36.420Z
Answers.LastActivityDate: 2020-10-18T08:43:36.420Z
Answers.UserDisplayName: Tom Dörr
Body: <p>I'm trying to have a simple autoencoder but with variable latent length (the network can produce variable latent lengths with respect to the complexity of the input), but I've not seen any related work to get idea from. Have you seen any related work? Do you have any idea to do so?</p>
  <p>Actually, I want to use this autoencoder for transmitting the data over a noisy channel, so having a variable-length may help.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-18T13:46:17.943Z
Comments.Text: What do you mean with  complexity of the input? What is your input? You mean something like texts and the input complexity is the number of words of your texts?
Comments.UserDisplayName: ddaedalus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-18T17:05:06.240Z
Comments.Text: it can be text and in that case the number of words is a good measure of complexity. I've seen an example of doing such a thing in [this](https://web.stanford.edu/~milind/papers/variablesc_spawc.pdf) paper but I think its not a good way to do so. I want a little more general idea, where the model can understand the complexity itself and determine the length of latent with respect to that
Comments.UserDisplayName: amin
Tags: <deep-learning><reference-request><autoencoders><latent-variable>
Title: Is it possible to have a variable-length latent vector in an autoencoder?
UserDisplayName: amin
ThreadText: # Is it possible to have a variable-length latent vector in an autoencoder?
  
  (Asked by: username_0 on 2020-10-18)
  
  I'm trying to have a simple autoencoder but with variable latent length (the network can produce variable latent lengths with respect to the complexity of the input), but I've not seen any related work to get idea from. Have you seen any related work? Do you have any idea to do so?
  
  Actually, I want to use this autoencoder for transmitting the data over a noisy channel, so having a variable-length may help.
  
  **Tags:** deep-learning, reference-request, autoencoders, latent-variable
  > **Comments:**
  > **username_1**: What do you mean with  complexity of the input? What is your input? You mean something like texts and the input complexity is the number of words of your texts?  
  > **username_0**: it can be text and in that case the number of words is a good measure of complexity. I've seen an example of doing such a thing in [this](https://web.stanford.edu/~milind/papers/variablesc_spawc.pdf) paper but I think its not a good way to do so. I want a little more general idea, where the model can understand the complexity itself and determine the length of latent with respect to that  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-10-18. Score: 1
  
  If you use RNNs, then I think the solution is to use padding (zero padding) with max sequence length (that is the max number of words in a text) in order to tell your model to skip the zeros when possible. In that way, your model will try to learn a good representation of your input with fixed size. If you do not know this dimension, a solution may be to grid search this hyperparameter.
  
  If you still want to exploit the dimensionality difference, maybe you can train different models with fixed size dimension of the representation dependently of the dimension of the input. That is, for example, use one for small, one for medium and one for large dimensions, but this should surely require to have a large and quite balanced initial dataset.
  
  Another idea could be to use the autoencoder with a fixed latent dimension. Then, do effective clustering on your samples using their latent representation, considering that similar representations should have similar dimensionality requirements (?). After that, you could train your initial dataset on k models, the same number as the clusters. That is, there should be k different latent spaces. The goal is to match each instance to the correct model. At first, you should train them all with each instance, but as the training progresses, you should maybe go with binary search for each instance in order it to find the correct model, assuming that there is total order in the measuring of the dimensionality requirements. Of course, this is just an idea, I don't know if it is going to be really helpful at all.
  
  > **Comments:**
  > **username_0**: Unfortunately I can't understand your first paragraph, do you mean zero-padding the latent? can you explain more?  
  > **username_1**: No! In the input space in order to deal with the incosistent dimensionality.  
  > **username_0**: Ok, but how it helps with variable-length latent vector? What I understood from this paragraph will help to have different length input (which RNNs can do with no problem at all)  
  > **username_0**: Your second paragraph is the way they have done in the paper I mentioned before (this one). But training a model for each latent length is not what I want, because I want the network (not me) to measure the complexity of input (useful when the input is image) and be able to produce any latent length (not those I predetermined)  
  > **username_1**: Check my post. I fixed it.  
  
  ---
  
  ### Answer by username_2 on 2020-10-18. Score: -1
  
  You might want to look at an encoder-decoder sequence to sequence model. This model allows you to input and output data with variable length.
  
  > **Comments:**
  > **username_0**: Maybe I asked my question in a bad way. The length of input or output doesn't matter to me (and they can be both fixed). But I want the network to produce variable length latent. Like if it see a complex image, it should use more latent units to reconstruct it, but if the image of the same size is simple (e.g. mostly black), the network should decide to use less latent units and it will be able to get almost the same reconstruction error.  
  
  ---
  
  