Answers.Body: <p>When an encoder outputs an embedded vector that is smaller than its input, the compression is lossy. The reversal process is not simple, it cannot be derived purely from analysis of input weights, but must be learned statistically from the data.</p>
  <p>The part that is mathematically irreversable and that prevents your idea working might not be what you think. It is due to the <em>addition</em> or summing of weighted values.</p>
  <p>Addition is only reversible (by subtraction) when you know the output sum and the sum of all the inputs other than the one you want to discover (e.g. for <span class="math-container">$4 + 6 = 10$</span> you can reverse it as <span class="math-container">$10 - 6 = 4$</span> if you know the output <span class="math-container">$10$</span> and the value <span class="math-container">$6$</span>). If you only know that two values <span class="math-container">$x + y$</span> sum to a specific value <span class="math-container">$z$</span>, this is not enough information to figure out what <span class="math-container">$x$</span> and <span class="math-container">$y$</span> are.</p>
  <p>In a neural network layer in the decoder, you know the sum but not any of the inputs. So the weighted sum that creates the logit of the embedding <span class="math-container">$y_j = \sum_i W_{ij}x_i + b_j$</span> cannot be reversed over even the simplest two-layer autoencoder, because the decoder doesn't have access to any of the <span class="math-container">$x_i$</span> values. Instead the decoder must take a statistical guess based on the training it has had, which has altered weights due to distribution and covariances of the inputs and the matching decodings.</p>
  <p>As an aside, for an autoencoder, you <em>can</em> set rules for the weights of the decoder to match or invert the weights of an encoder. However, those cannot then be <em>arbitrary</em> weights, but the neural network training will have to try to find a set of weights that both meet this constraint <em>and</em> result in statistically viable outputs. Forcing this kind of weight matching is a form of regularisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T22:18:51.510Z
Answers.LastActivityDate: 2024-01-31T22:30:20.663Z
Answers.LastEditDate: 2024-01-31T22:30:20.663Z
Answers.UserDisplayName: Neil Slater
Body: <p>If you have an encoder decoder with 10 input neurons for X then 3 hidden in one layer then another 10 in the output which are the same X is it not trivial to set the weights whatever you want and w1 to w30 and then make the weights from the hidden to the output 1/w1,1/w2,...,1/w30.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-31T09:21:29.387Z
Comments.Text: Well you could try your idea - trivially, as you suggest (perhaps 15 minutes to code). What happens? Try for some random values of X. How many are decoded correctly?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-31T17:06:32.420Z
Comments.Text: It should not make sense mathematically (without the code), but I do not know what stops it from making sense.
Comments.UserDisplayName: J_Bake
Tags: <neural-networks><autoencoders><encoder-decoder>
Title: Why aren't encoders decoders trivial?
UserDisplayName: J_Bake
ThreadText: # Why aren't encoders decoders trivial?
  
  (Asked by: username_0 on 2024-01-31)
  
  If you have an encoder decoder with 10 input neurons for X then 3 hidden in one layer then another 10 in the output which are the same X is it not trivial to set the weights whatever you want and w1 to w30 and then make the weights from the hidden to the output 1/w1,1/w2,...,1/w30.
  
  **Tags:** neural-networks, autoencoders, encoder-decoder
  
  > **Comments:**
  > **username_1**: Well you could try your idea - trivially, as you suggest (perhaps 15 minutes to code). What happens? Try for some random values of X. How many are decoded correctly?  
  > **username_0**: It should not make sense mathematically (without the code), but I do not know what stops it from making sense.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-31. Score: 2
  
  When an encoder outputs an embedded vector that is smaller than its input, the compression is lossy. The reversal process is not simple, it cannot be derived purely from analysis of input weights, but must be learned statistically from the data.
  
  The part that is mathematically irreversable and that prevents your idea working might not be what you think. It is due to the *addition* or summing of weighted values.
  
  Addition is only reversible (by subtraction) when you know the output sum and the sum of all the inputs other than the one you want to discover (e.g. for $4 + 6 = 10$ you can reverse it as $10 - 6 = 4$ if you know the output $10$ and the value $6$). If you only know that two values $x + y$ sum to a specific value $z$, this is not enough information to figure out what $x$ and $y$ are.
  
  In a neural network layer in the decoder, you know the sum but not any of the inputs. So the weighted sum that creates the logit of the embedding $y\_j = \sum\_i W\_{ij}x\_i + b\_j$ cannot be reversed over even the simplest two-layer autoencoder, because the decoder doesn't have access to any of the $x\_i$ values. Instead the decoder must take a statistical guess based on the training it has had, which has altered weights due to distribution and covariances of the inputs and the matching decodings.
  
  As an aside, for an autoencoder, you *can* set rules for the weights of the decoder to match or invert the weights of an encoder. However, those cannot then be *arbitrary* weights, but the neural network training will have to try to find a set of weights that both meet this constraint *and* result in statistically viable outputs. Forcing this kind of weight matching is a form of regularisation.
  
  
  ---
  
  