Answers.Body: <p>The meaning of <em>invertible</em> here is the standard definition of invertibility for a mathematical function <span class="math-container">$f \colon X \to Y$</span>. Invertible simply means &quot;the function has an inverse map <span class="math-container">$f^{-1} \colon Y \to X$</span>&quot;. Equivalently the function <span class="math-container">$f$</span> is <em>bijective</em>, which means the following two conditions hold:</p>
  <ol>
  <li><p><span class="math-container">$f$</span> is <em>injective</em>: for any two distinct <span class="math-container">$x_1, x_2 \in X$</span>, <span class="math-container">$f(x_1) \ne f(x_2)$</span>.</p>
  </li>
  <li><p><span class="math-container">$f$</span> is <em>surjective</em>: for any <span class="math-container">$y \in Y$</span>, there exists an <span class="math-container">$x \in X$</span> such that <span class="math-container">$f(x) = y$</span>.</p>
  </li>
  </ol>
  <p>If this is unfamiliar, you should be able to find some helpful references on Google using these terms.</p>
  <p>Most obvious neural network architectures cannot possibly be invertible. Consider for example a classifier which takes an image or some other high-dimensional input, and outputs a classification label. This network could only be invertible if there was only one possible input<sup>1</sup> which corresponds to each label, which is not the goal of the network.</p>
  <hr />
  <p><sup>1</sup> I mean this rather literally: if one pixel is even slightly different, this would be a different input to the network and would have to have a different output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T09:23:15.537Z
Answers.Text: Thank you so much that explains it wonderfully!  The last point you referred to is exactly what I have been investigating how to invert a high dimensional input for a classification problem. Is there any literature you can refer me to which states that in such cases the invertibility is not possible?
Answers.UserDisplayName: user157522
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T09:32:58.940Z
Answers.Text: @user157522 Glad to hear that. That fact follows from the definition of invertibility, so it might not be stated explicitly in the literature in the context of neural networks. You can probably prove it to yourself after thinking about the definitions (perhaps take a read through the [Wikipedia article](https://en.wikipedia.org/wiki/Bijection) as a start). In particular there is no way to form an injective map from an infinite set to a finite one (you can find this in any text on set theory, or search for this result directly), which more or less settles the question for a classifier.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T09:56:34.937Z
Answers.Text: Thanks a lot, I understood it, but I kind of have to provide literature that explicitly states that to prove my point.
Answers.UserDisplayName: user157522
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T10:21:32.940Z
Answers.Text: @user157522 Okay, it might be mentioned in passing in a paper on invertible networks, but I don't think the result will be particularly prominent unfortunately. Depending on your audience, assuming a knowledge of elementary concepts like bijections might be reasonable (although of course what is elementary to a mathematician who works with these concepts all day might not be elementary to someone else!). Perhaps giving a brief proof of the result would be sufficient, with references to a mathematical text for readers who would be interested?
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-29T22:01:38.863Z
Answers.Text: Excellent answer. For @user157522, you may be interested to check out https://hci.iwr.uni-heidelberg.de/vislearn/inverse-problems-invertible-neural-networks/ and https://github.com/jaekookang/invertible_neural_networks. And a recent example from my field: https://www.science.org/doi/10.1126/science.aaw1147
Answers.UserDisplayName: R. Barrett
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T09:11:30.047Z
Answers.LastActivityDate: 2021-04-20T09:11:30.047Z
Answers.UserDisplayName: htl
Body: <p>I am interested in learning about the inverse of neural networks and I would like to understand about the invertibility of neural networks, as for example described in <a href="https://openreview.net/forum?id=BJlVeyHFwH" rel="nofollow noreferrer"><em>On the Invertibility of Invertible Neural Networks</em></a>.</p>
  <p>Researchers who are working on this domain, can you help me understand these two questions.</p>
  <ol>
  <li>Are all neural network invertible ?</li>
  <li>What exactly qualifies a neural network to be invertible ?</li>
  </ol>
  
Tags: <neural-networks>
Title: Are neural networks invertible?
UserDisplayName: user157522
ThreadText: # Are neural networks invertible?
  
  (Asked by: username_0 on 2021-04-20)
  
  I am interested in learning about the inverse of neural networks and I would like to understand about the invertibility of neural networks, as for example described in [*On the Invertibility of Invertible Neural Networks*](https://openreview.net/forum?id=BJlVeyHFwH).
  
  Researchers who are working on this domain, can you help me understand these two questions.
  
  1. Are all neural network invertible ?
  2. What exactly qualifies a neural network to be invertible ?
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-20. Score: 3 | Accepted answer
  
  The meaning of *invertible* here is the standard definition of invertibility for a mathematical function $f \colon X \to Y$. Invertible simply means "the function has an inverse map $f^{-1} \colon Y \to X$". Equivalently the function $f$ is *bijective*, which means the following two conditions hold:
  
  1. $f$ is *injective*: for any two distinct $x\_1, x\_2 \in X$, $f(x\_1) \ne f(x\_2)$.
  2. $f$ is *surjective*: for any $y \in Y$, there exists an $x \in X$ such that $f(x) = y$.
  
  If this is unfamiliar, you should be able to find some helpful references on Google using these terms.
  
  Most obvious neural network architectures cannot possibly be invertible. Consider for example a classifier which takes an image or some other high-dimensional input, and outputs a classification label. This network could only be invertible if there was only one possible input1 which corresponds to each label, which is not the goal of the network.
  
  ---
  
  1 I mean this rather literally: if one pixel is even slightly different, this would be a different input to the network and would have to have a different output.
  
  > **Comments:**
  > **username_0**: Thank you so much that explains it wonderfully!  The last point you referred to is exactly what I have been investigating how to invert a high dimensional input for a classification problem. Is there any literature you can refer me to which states that in such cases the invertibility is not possible?  
  > **username_1**: @username_0 Glad to hear that. That fact follows from the definition of invertibility, so it might not be stated explicitly in the literature in the context of neural networks. You can probably prove it to yourself after thinking about the definitions (perhaps take a read through the [Wikipedia article](https://en.wikipedia.org/wiki/Bijection) as a start). In particular there is no way to form an injective map from an infinite set to a finite one (you can find this in any text on set theory, or search for this result directly), which more or less settles the question for a classifier.  
  > **username_0**: Thanks a lot, I understood it, but I kind of have to provide literature that explicitly states that to prove my point.  
  > **username_1**: @username_0 Okay, it might be mentioned in passing in a paper on invertible networks, but I don't think the result will be particularly prominent unfortunately. Depending on your audience, assuming a knowledge of elementary concepts like bijections might be reasonable (although of course what is elementary to a mathematician who works with these concepts all day might not be elementary to someone else!). Perhaps giving a brief proof of the result would be sufficient, with references to a mathematical text for readers who would be interested?  
  > **username_2**: Excellent answer. For @username_0, you may be interested to check out https://hci.iwr.uni-heidelberg.de/vislearn/inverse-problems-invertible-neural-networks/ and https://github.com/jaekookang/invertible_neural_networks. And a recent example from my field: https://www.science.org/doi/10.1126/science.aaw1147  
  
  ---
  
  