Body: <p>I'm fairly new to deep learning and looking for some reference literature... Specifically, I want to train a neural network to predict vectors <span class="math-container">$v \in \mathbb{R}^3$</span> under the constraint <span class="math-container">$||v||\leq 1$</span>.</p>
  <p>I add that the case <span class="math-container">$v\simeq0$</span> is the most common in my dataset. Which final layer(s) would you suggest to accomplish this? These layers would follow a CNN + fully connected layers body.</p>
  <p>Thanks in advance</p>
  <hr />
  <p>Background: this vector would represent the <span class="math-container">$SO(3)$</span> element of a rotation matrix</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-28T21:35:33.950Z
Comments.Text: Is the constraint only that you *know* the target dataset has nothing with $||v|| \gt 1$, or are there negative consequences for predicting magnitudes that are even slightly too large? How common is $||v|| = 1$ (lots of stuff on the boundary as well as near the origin?). What is your loss function? If you are not sure of the loss function, what are the consequences for inaccurate predictions and do they vary according to absolute error distance, ratio of prediction to true value or some other factor?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T08:17:26.383Z
Comments.Text: At $||v||=1$ a discontinuity in the loss function will happen. Basically, $cn$ and $(c+2k\pi)n$ are equivalent, with $||n||=1$, because they represent the same rotation vector (the constraint is actually $||v||=\pi$, but I put $1$ in the question for simplicity.
Comments.UserDisplayName: Lilla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T08:19:01.823Z
Comments.Text: $||v||=1$ (actually, $||v||=2\pi$) will happen around 1-5 % of the times, whereas we're near the origin 60% of the times (meaning little rotation)
Comments.UserDisplayName: Lilla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T08:25:06.330Z
Comments.Text: As you are asking about output representation you need to be clear how you want the raw output to be scaled. When considering unscaled rotation you have an interesting issue that $||v|| = 2\pi$ is equivalent to  $||v|| = 0$. As you are considering opposite rotations, then you could equally scale to $||v|| = pi$ as there is a discontinuity of sorts there anyway (when $||v|| = \pi$, then $v \equiv -v$) and you get full coverage of all possible new orientations under the constraint $||v|| \le \pi$
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T08:29:04.713Z
Comments.Text: ... Unless the rotations are defining the act of rotating as opposed to the final orientation? But in that case, it is not clear why you would stop at one complete revolution.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T08:54:26.723Z
Comments.Text: No, I really want to regress rotations. The problem is that I would like the rotation representation to be smooth, connected and easily invertible, but there is no such thing as this, apparently, unless I regress a full $9\times 9$ matrix, but then there is a problem of enforcing the orthogonality constraint, and I can't easily work with outputs on a non euclidean manifold (because of how I'll have to work with this NN).
Comments.UserDisplayName: Lilla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T09:54:04.113Z
Comments.Text: The issue as I understand your problem is not the constraint. A predicted value that does not repsect the constraint, but that is close to the true value would still be good. The problem is that you have input data with similar inputs that should predict either $-v$ or $+v$ around 1-5% of the time, and the NN may have trouble learning this discontinuity. There is a good chance it will learn to predict close to zero rotation in cases where you want close to maximum rotation. You will be better off with a representation that does not have the discontinuity
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T12:46:30.870Z
Comments.Text: @Leonardo "I can't easily work with outputs on a non euclidean manifold"  maybe this is a silly suggestion and you already know about it, you could check [geotorch](https://github.com/Lezcano/geotorch)
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-29T13:26:44.203Z
Comments.Text: @EdoardoGuerriero Didn't know about this, looks very interesting!
Comments.UserDisplayName: Lilla
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-30T14:33:05.963Z
Comments.Text: Can you please put your **specific question** in the title? It seems that your specific question is "Which final layer(s) can I use to produce 3d vectors such that their norm is less than one (an open ball)?". I would also recommend that you update your post with more details about your specific problem (e.g. what high-level task are you really trying to solve? in which context?) and with the info that you've gathered from the discussions in the comments. I would also describe more in detail your dataset. For example, what are the inputs? How many data points?
Comments.UserDisplayName: nbro
Tags: <deep-learning><objective-functions>
Title: Learning values in open ball: which final layers to employ?
UserDisplayName: Lilla
ThreadText: # Learning values in open ball: which final layers to employ?
  
  (Asked by: username_0 on 2022-04-28)
  
  I'm fairly new to deep learning and looking for some reference literature... Specifically, I want to train a neural network to predict vectors $v \in \mathbb{R}^3$ under the constraint $||v||\leq 1$.
  
  I add that the case $v\simeq0$ is the most common in my dataset. Which final layer(s) would you suggest to accomplish this? These layers would follow a CNN + fully connected layers body.
  
  Thanks in advance
  
  ---
  
  Background: this vector would represent the $SO(3)$ element of a rotation matrix
  
  **Tags:** deep-learning, objective-functions
  
  > **Comments:**
  > **username_1**: Is the constraint only that you *know* the target dataset has nothing with $||v|| \gt 1$, or are there negative consequences for predicting magnitudes that are even slightly too large? How common is $||v|| = 1$ (lots of stuff on the boundary as well as near the origin?). What is your loss function? If you are not sure of the loss function, what are the consequences for inaccurate predictions and do they vary according to absolute error distance, ratio of prediction to true value or some other factor?  
  > **username_0**: At $||v||=1$ a discontinuity in the loss function will happen. Basically, $cn$ and $(c+2k\pi)n$ are equivalent, with $||n||=1$, because they represent the same rotation vector (the constraint is actually $||v||=\pi$, but I put $1$ in the question for simplicity.  
  > **username_0**: $||v||=1$ (actually, $||v||=2\pi$) will happen around 1-5 % of the times, whereas we're near the origin 60% of the times (meaning little rotation)  
  > **username_1**: As you are asking about output representation you need to be clear how you want the raw output to be scaled. When considering unscaled rotation you have an interesting issue that $||v|| = 2\pi$ is equivalent to  $||v|| = 0$. As you are considering opposite rotations, then you could equally scale to $||v|| = pi$ as there is a discontinuity of sorts there anyway (when $||v|| = \pi$, then $v \equiv -v$) and you get full coverage of all possible new orientations under the constraint $||v|| \le \pi$  
  > **username_1**: ... Unless the rotations are defining the act of rotating as opposed to the final orientation? But in that case, it is not clear why you would stop at one complete revolution.  
  > **username_0**: No, I really want to regress rotations. The problem is that I would like the rotation representation to be smooth, connected and easily invertible, but there is no such thing as this, apparently, unless I regress a full $9\times 9$ matrix, but then there is a problem of enforcing the orthogonality constraint, and I can't easily work with outputs on a non euclidean manifold (because of how I'll have to work with this NN).  
  > **username_1**: The issue as I understand your problem is not the constraint. A predicted value that does not repsect the constraint, but that is close to the true value would still be good. The problem is that you have input data with similar inputs that should predict either $-v$ or $+v$ around 1-5% of the time, and the NN may have trouble learning this discontinuity. There is a good chance it will learn to predict close to zero rotation in cases where you want close to maximum rotation. You will be better off with a representation that does not have the discontinuity  
  > **username_2**: @Leonardo "I can't easily work with outputs on a non euclidean manifold"  maybe this is a silly suggestion and you already know about it, you could check [geotorch](https://github.com/Lezcano/geotorch)  
  > **username_0**: @username_2 Didn't know about this, looks very interesting!  
  > **username_3**: Can you please put your **specific question** in the title? It seems that your specific question is "Which final layer(s) can I use to produce 3d vectors such that their norm is less than one (an open ball)?". I would also recommend that you update your post with more details about your specific problem (e.g. what high-level task are you really trying to solve? in which context?) and with the info that you've gathered from the discussions in the comments. I would also describe more in detail your dataset. For example, what are the inputs? How many data points?  
  
  ---
  
  ## 0 answers
  
  