Answers.Body: <p>The main problem with simply using the values <span class="math-container">$\alpha \in [0, 2\pi]$</span> is that semantically <span class="math-container">$0 = 2\pi$</span>, but numerically <span class="math-container">$0$</span> and <span class="math-container">$2\pi$</span> are maximally far apart. A common way to encode this is by a vector of <span class="math-container">$\sin$</span> and <span class="math-container">$\cos$</span>. It perfectly conveys the fact that <span class="math-container">$0 = 2\pi$</span>, because:</p>
  <p><span class="math-container">$$
  \begin{bmatrix}
      \sin(0)\\
      \cos(0)
  \end{bmatrix}
  = 
  \begin{bmatrix}
      \sin(2\pi)\\
      \cos(2\pi)
  \end{bmatrix}
  $$</span></p>
  <p>This encoding essentially maps the angle values onto the 2D unit circle. In order to decode this, you can calculate
  <span class="math-container">$$\text{atan}2(a_1, a_2) = \alpha,$$</span></p>
  <p>where <span class="math-container">$a_1 = \sin(\alpha)$</span> and <span class="math-container">$a_2 = \cos(\alpha)$</span>.</p>
  <p><a href="https://www.avanwyk.com/encoding-cyclical-features-for-deep-learning/" rel="noreferrer">Here</a> is a nice detailed explanation and here are two references, where this is applied:</p>
  <ul>
  <li><a href="https://academic.oup.com/bioinformatics/article/33/18/2842/3738544?login=false#118817095" rel="noreferrer">Heffernan et al.</a></li>
  <li><a href="https://arxiv.org/abs/2103.06879" rel="noreferrer">Pizzati et al.</a></li>
  </ul>
  <p><strong>EDIT</strong> As it was noted in the comments: The values <span class="math-container">$\sin(\alpha)$</span> and <span class="math-container">$\cos(\alpha)$</span> are not independent and the following naturally holds: <span class="math-container">$\sqrt{\sin(\alpha)^2 + \cos(\alpha)^2}= 1$</span>, i.e. the euclidean norm is one. In a situation where your Neural Network predicts the sin and cos values, this condition isn't necessarily true. Therefore, you should consider adding a regularization term to the loss that guides the neural network toward outputting valid values (with unit norm) which could look like this:</p>
  <p><span class="math-container">$$
  r_\lambda\left(\hat{y}_1, \hat{y}_2\right)\; = \lambda \left(\; 1 - \sqrt{\hat{y}_1^2 + \hat{y}_2^2}\right),
  $$</span></p>
  <p>where <span class="math-container">$\hat{y}_1$</span> and <span class="math-container">$\hat{y}_2$</span> are the sin and cos outputs of the network respectively and <span class="math-container">$\lambda$</span> is a scalar that weights the regularization term against the loss. I found <a href="https://arxiv.org/abs/1805.06485" rel="noreferrer">this</a> paper where such a regularization term is used (s. Sec. 3.2) to get valid quaternions (Quaternions must also have unit norm). They found that many values work for <span class="math-container">$\lambda$</span> and they settle for <span class="math-container">$\lambda = 0.1$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T13:12:20.223Z
Answers.Text: Do you know if there is a general approach for that? The closest thing that comes to my mind are group convolutional networks. Personally though, I would prefer a simpler, less number crunch heavy method
Answers.UserDisplayName: Imago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T13:31:26.317Z
Answers.Text: I'm not sure what exactly you are looking for when you ask for a general approach - on the one hand, there is general angular data (which you can encode with the method described). And then there is the context of those values (i.e. the domain). If the domain involves rotational symmetry then you should look into group equivariant convolutional networks (which is probably what you mean by group convolutional network, right?). Without knowing the domain, it is not clear if group equivariant convolutions are a reasonable thing to use. What is the dataset you are using?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T15:23:59.090Z
Answers.Text: You only need to use $sin(\alpha)$ (or $ccos(\alpha)$) as the label, right?  No need (and undesirable) to have both as multilabel outputs given their strict dependence.
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T16:45:21.953Z
Answers.Text: Actually, you should use both otherwise you would get singularities, e.g. for $\sin$ this would happen at $\sin(0) = \sin(\pi) = \sin(2\pi)$. Whereas, if you encode it as the tuple, you get a unique vector for all $\alpha \in [0, 2\pi)$ with the property that at $0$ and $2\pi$ the encoding is the same. You are right that the values are dependent. I've often seen that there is a regularization term in NN training like this: $\parallel [x_1, x_2]^T \parallel_2 = 1$, which guides the network toward outputting valid numbers for the two outputs $x_1, x_2$ where the norm is one.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T17:02:53.500Z
Answers.Text: Thanks for pointing that out @SnehalPatel, I added the regularization thing to the answer
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-30T14:57:04.620Z
Answers.Text: @Chillston I didn't have a specific data set in set. I was wondering how one would handle the case, when the given data has some symmetries, structure or invariances towards specific transformation. In case of angular data as above, a neural network can learn to treat the regions are around an angle 0 and angle 2pi equally, however it is not guaranteed that the network will pick up on that structure. So, how can one simplify this process in general, as done with the angle problem above?
Answers.UserDisplayName: Imago
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-01T12:49:33.890Z
Answers.Text: @Imago The general form of that would be group-specific neural networks. So when you have a symmetry w.r.t a specific group, then you can design neural operations that obey these symmetries. For 2D rotation invariant data, you'd want a kernel that itself is invariant to the orthogonal group O(2). However, this is a different concept to what the question is about. Proper angle encoding doesn't necessarily mean that you have rotational symmetry. Maybe you find the [protobook about Geometric Deep Learning](https://geometricdeeplearning.com/) a good read (or you already know it).
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-27T11:51:41.537Z
Answers.LastActivityDate: 2022-11-27T20:49:00.277Z
Answers.LastEditDate: 2022-11-27T20:49:00.277Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>You might want to look at the von Mises Distribution, it defines a probability distribution over angles.</p>
  <p>See <a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf" rel="nofollow noreferrer">Pattern Recognition and Machine Learning</a>, Christopher Bishop, Appendix B (pg 693), or alternatively <a href="https://en.wikipedia.org/wiki/Von_Mises_distribution" rel="nofollow noreferrer">Wikipedia</a> has an article on this.</p>
  <p>You could certainly use this as a loss function in a neural network. My only reservation is that the input parameter <span class="math-container">$\theta_o$</span> is itself periodic which might not play well with standard neural network architectures? Therefore previous answers are also worth considering.</p>
  <p>I only mention this as if you are interested in distributions over angles, the von Mises distribution is something you should probably be aware of.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T14:47:42.113Z
Answers.Text: I'm not sure I understand what you mean by "You could certainly use this as a loss function in a neural network?"  Can you provide an explanation of how it the probability density function can be used as a loss function?
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T15:15:34.173Z
Answers.Text: Von Mises distribution defines a probability density. You take the logarithm of this function and then take the negative of that; that is the loss function.   This provides what I think is a pretty good explanation of the basic approach:
  [link](https://goodboychan.github.io/python/coursera/tensorflow_probability/icl/2021/08/19/01-Maximum-likelihood-estimation.html#The-negative-log-likelihood)
Answers.UserDisplayName: Julian Francis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T11:35:38.700Z
Answers.LastActivityDate: 2022-11-28T16:07:36.680Z
Answers.LastEditDate: 2022-11-28T16:07:36.680Z
Answers.UserDisplayName: Julian Francis
Body: <p>I am training a neural network where the target data is a vector of angles in radians (between <span class="math-container">$0$</span> and <span class="math-container">$2\pi$</span>).</p>
  <p>I am looking for study material on how to encode this data.</p>
  <p>Can you supply me with a book or research paper that covers this topic comprehensively?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-27T22:19:57.253Z
Comments.Text: Many suggestions here: https://stats.stackexchange.com/questions/565038/why-doesnt-mean-square-error-work-in-case-of-angular-data/565057#565057
Comments.UserDisplayName: Sycorax
Tags: <neural-networks><reference-request><training><datasets><data-preprocessing>
Title: How can I encode angle data to train neural networks?
UserDisplayName: user366312
ThreadText: # How can I encode angle data to train neural networks?
  
  (Asked by: username_0 on 2022-11-27)
  
  I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$).
  
  I am looking for study material on how to encode this data.
  
  Can you supply me with a book or research paper that covers this topic comprehensively?
  
  **Tags:** neural-networks, reference-request, training, datasets, data-preprocessing
  > **Comments:**
  > **username_1**: Many suggestions here: https://stats.stackexchange.com/questions/565038/why-doesnt-mean-square-error-work-in-case-of-angular-data/565057#565057  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-11-27. Score: 17
  
  The main problem with simply using the values $\alpha \in [0, 2\pi]$ is that semantically $0 = 2\pi$, but numerically $0$ and $2\pi$ are maximally far apart. A common way to encode this is by a vector of $\sin$ and $\cos$. It perfectly conveys the fact that $0 = 2\pi$, because:
  
  $$
  \begin{bmatrix}
  \sin(0)\\
  \cos(0)
  \end{bmatrix}
  =
  \begin{bmatrix}
  \sin(2\pi)\\
  \cos(2\pi)
  \end{bmatrix}
  $$
  
  This encoding essentially maps the angle values onto the 2D unit circle. In order to decode this, you can calculate
  $$\text{atan}2(a\_1, a\_2) = \alpha,$$
  
  where $a\_1 = \sin(\alpha)$ and $a\_2 = \cos(\alpha)$.
  
  [Here](https://www.avanwyk.com/encoding-cyclical-features-for-deep-learning/) is a nice detailed explanation and here are two references, where this is applied:
  
  * [Heffernan et al.](https://academic.oup.com/bioinformatics/article/33/18/2842/3738544?login=false#118817095)
  * [Pizzati et al.](https://arxiv.org/abs/2103.06879)
  
  **EDIT** As it was noted in the comments: The values $\sin(\alpha)$ and $\cos(\alpha)$ are not independent and the following naturally holds: $\sqrt{\sin(\alpha)^2 + \cos(\alpha)^2}= 1$, i.e. the euclidean norm is one. In a situation where your Neural Network predicts the sin and cos values, this condition isn't necessarily true. Therefore, you should consider adding a regularization term to the loss that guides the neural network toward outputting valid values (with unit norm) which could look like this:
  
  $$
  r\_\lambda\left(\hat{y}\_1, \hat{y}\_2\right)\; = \lambda \left(\; 1 - \sqrt{\hat{y}\_1^2 + \hat{y}\_2^2}\right),
  $$
  
  where $\hat{y}\_1$ and $\hat{y}\_2$ are the sin and cos outputs of the network respectively and $\lambda$ is a scalar that weights the regularization term against the loss. I found [this](https://arxiv.org/abs/1805.06485) paper where such a regularization term is used (s. Sec. 3.2) to get valid quaternions (Quaternions must also have unit norm). They found that many values work for $\lambda$ and they settle for $\lambda = 0.1$
  
  > **Comments:**
  > **username_3**: Do you know if there is a general approach for that? The closest thing that comes to my mind are group convolutional networks. Personally though, I would prefer a simpler, less number crunch heavy method  
  > **username_2**: I'm not sure what exactly you are looking for when you ask for a general approach - on the one hand, there is general angular data (which you can encode with the method described). And then there is the context of those values (i.e. the domain). If the domain involves rotational symmetry then you should look into group equivariant convolutional networks (which is probably what you mean by group convolutional network, right?). Without knowing the domain, it is not clear if group equivariant convolutions are a reasonable thing to use. What is the dataset you are using?  
  > **username_4**: You only need to use $sin(\alpha)$ (or $ccos(\alpha)$) as the label, right?  No need (and undesirable) to have both as multilabel outputs given their strict dependence.  
  > **username_2**: Actually, you should use both otherwise you would get singularities, e.g. for $\sin$ this would happen at $\sin(0) = \sin(\pi) = \sin(2\pi)$. Whereas, if you encode it as the tuple, you get a unique vector for all $\alpha \in [0, 2\pi)$ with the property that at $0$ and $2\pi$ the encoding is the same. You are right that the values are dependent. I've often seen that there is a regularization term in NN training like this: $\parallel [x_1, x_2]^T \parallel_2 = 1$, which guides the network toward outputting valid numbers for the two outputs $x_1, x_2$ where the norm is one.  
  > **username_2**: Thanks for pointing that out @SnehalPatel, I added the regularization thing to the answer  
  > **username_3**: @Chillston I didn't have a specific data set in set. I was wondering how one would handle the case, when the given data has some symmetries, structure or invariances towards specific transformation. In case of angular data as above, a neural network can learn to treat the regions are around an angle 0 and angle 2pi equally, however it is not guaranteed that the network will pick up on that structure. So, how can one simplify this process in general, as done with the angle problem above?  
  > **username_2**: @Imago The general form of that would be group-specific neural networks. So when you have a symmetry w.r.t a specific group, then you can design neural operations that obey these symmetries. For 2D rotation invariant data, you'd want a kernel that itself is invariant to the orthogonal group O(2). However, this is a different concept to what the question is about. Proper angle encoding doesn't necessarily mean that you have rotational symmetry. Maybe you find the [protobook about Geometric Deep Learning](https://geometricdeeplearning.com/) a good read (or you already know it).  
  
  ---
  
  ### Answer by username_5 on 2022-11-28. Score: 0
  
  You might want to look at the von Mises Distribution, it defines a probability distribution over angles.
  
  See [Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf), Christopher Bishop, Appendix B (pg 693), or alternatively [Wikipedia](https://en.wikipedia.org/wiki/Von_Mises_distribution) has an article on this.
  
  You could certainly use this as a loss function in a neural network. My only reservation is that the input parameter $\theta\_o$ is itself periodic which might not play well with standard neural network architectures? Therefore previous answers are also worth considering.
  
  I only mention this as if you are interested in distributions over angles, the von Mises distribution is something you should probably be aware of.
  
  > **Comments:**
  > **username_4**: I'm not sure I understand what you mean by "You could certainly use this as a loss function in a neural network?"  Can you provide an explanation of how it the probability density function can be used as a loss function?  
  > **username_5**: Von Mises distribution defines a probability density. You take the logarithm of this function and then take the negative of that; that is the loss function.   This provides what I think is a pretty good explanation of the basic approach: [link](https://goodboychan.github.io/python/coursera/tensorflow_probability/icl/2021/08/19/01-Maximum-likelihood-estimation.html#The-negative-log-likelihood)  
  
  ---
  
  