Answers.Body: <p>There are two basic problems with your code:</p>
  <ul>
  <li><p>The functions <code>sin(t)</code> and <code>cos(t)</code> (both in numpy and tensorflow) take radians as inputs. Seeing <code>Constant(90)</code> in your code, and the learning rate of <code>1.</code> I'm guessing that you assume that it is in degrees - that's incorrect.</p>
  </li>
  <li><p>In your training data <code>y_ta</code> <strong>is not a rotation</strong> of <code>y_in</code>:</p>
  <pre><code>y_in = np.array([np.sin(t), np.cos(t)]).T
  y_ta = np.array([np.cos(t), np.sin(t)]).T
  </code></pre>
  </li>
  </ul>
  <p>It is a reflection about <code>y=x</code> diagonal. No wonder it fails to find an appropriate rotation.</p>
  <p>I just had to change <code>y_ta</code> to:</p>
  <pre><code>  y_ta = np.array([-np.cos(t), np.sin(t)]).T
  </code></pre>
  <p>And train with more sensible learning rate:</p>
  <pre><code>  model.compile(tf.keras.optimizers.SGD(lr=0.1), loss='MSE')
    model.fit(y_in, y_ta, epochs=10)
  </code></pre>
  <p>To get the angle (which I then convert to degrees):</p>
  <pre><code>  (180 * model.layers[0].kernel[0,0].numpy() / np.pi) % 360
    &gt; 90.00187079311581
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-21T09:11:08.373Z
Answers.Text: Thanks and yes, I believe I messed up with the target value which was not actually a rotation.
Answers.UserDisplayName: damith219
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-20T14:46:17.760Z
Answers.LastActivityDate: 2021-04-20T14:46:17.760Z
Answers.UserDisplayName: Kostya
Body: <p>I'm trying to implement a neural network that can capture the drift in a measured angle as a way of dynamic calibration. i.e, I have a reference system that may change throughout the course of the data gathering and would like to train a network layer which actually converts the drifting reference to the desired reference by updating the angle parameter.</p>
  <p>For example: Consider the 2d case. We would have a set of 2d points <span class="math-container">$X\in \mathbb{R}^2$</span> and a trainable parameter called <span class="math-container">$\theta$</span> in the layer. The output of the layer would then be:
  <span class="math-container">$$X_o = XR$$</span> where
  <span class="math-container">$$R = \begin{bmatrix}
  \cos(\theta) &amp; -\sin(\theta) \\ 
  \sin(\theta) &amp; \cos(\theta)
  \end{bmatrix}$$</span></p>
  <p>Using Adam optimizer I then try to find the <span class="math-container">$\theta$</span> which transforms a given angle to the desired reference.</p>
  <p>However, the <span class="math-container">$\theta$</span> value seems to fluctuate around the initial value probably because of a diverging gradient(?). How can I overcome this issue?</p>
  <p>The code is below.</p>
  <pre><code>import tensorflow as tf
  import numpy as np
  import matplotlib.pyplot as plt
  
  
  class Rotation2D(tf.keras.layers.Layer):
    def __init__(self):
      super(Rotation2D, self).__init__()
  
    def build(self, input_shape):
      self.kernel = self.add_weight(&quot;kernel&quot;, initializer=tf.keras.initializers.Constant(90),
                                    shape=[1, 1])
  
    def call(self, input):
      matrix = ([[tf.cos(self.kernel[0, 0]), -tf.sin(self.kernel[0, 0])],
                [tf.sin(self.kernel[0, 0]), tf.cos(self.kernel[0, 0])]])
      return tf.matmul(input, tf.transpose(matrix))
  
  layer = Rotation2D()
  
  t = np.arange(0, 1000)/200.
  
  y_in = np.array([np.sin(t), np.cos(t)]).T
  y_ta = np.array([np.cos(t), np.sin(t)]).T
  
  model = tf.keras.Sequential()
  model.add(layer)
  
  model.compile(tf.keras.optimizers.SGD(lr=1.), loss='MSE')
  model.fit(y_in, y_ta, epochs=1)
  for i in range(100):
    print(layer.get_weights())
    model.fit(y_in, y_ta,verbose=0, batch_size=5)
  y_out = (model.predict(y_in))
  
  fig, axes = plt.subplots(2, 1)
  
  for i in range(2):
    ax = axes[i]
  
    ax.plot(y_in.T[i], label = 'input')
    ax.plot(y_ta.T[i], label = 'target')
    ax.plot(y_out.T[i], label = 'prediction')
  
  plt.legend()
  
  plt.show()<span class="math-container">```</span>
  </code></pre>
  
Tags: <deep-neural-networks>
Title: Unable to 'learn' a rotational angle by parametrising the angle as a neural network layer
UserDisplayName: damith219
ThreadText: # Unable to 'learn' a rotational angle by parametrising the angle as a neural network layer
  
  (Asked by: username_0 on 2021-04-20)
  
  I'm trying to implement a neural network that can capture the drift in a measured angle as a way of dynamic calibration. i.e, I have a reference system that may change throughout the course of the data gathering and would like to train a network layer which actually converts the drifting reference to the desired reference by updating the angle parameter.
  For example: Consider the 2d case. We would have a set of 2d points $X\in \mathbb{R}^2$ and a trainable parameter called $\theta$ in the layer. The output of the layer would then be:
  $$X_o = XR$$ where
  $$R = \begin{bmatrix}
  \cos(\theta) & -\sin(\theta) \\ 
  \sin(\theta) & \cos(\theta)
  \end{bmatrix}$$
  Using Adam optimizer I then try to find the $\theta$ which transforms a given angle to the desired reference.
  However, the $\theta$ value seems to fluctuate around the initial value probably because of a diverging gradient(?). How can I overcome this issue?
  The code is below.
  import tensorflow as tf
  import numpy as np
  import matplotlib.pyplot as plt
  
  
  class Rotation2D(tf.keras.layers.Layer):
    def __init__(self):
      super(Rotation2D, self).__init__()
  
    def build(self, input_shape):
      self.kernel = self.add_weight("kernel", initializer=tf.keras.initializers.Constant(90),
                                    shape=[1, 1])
  
    def call(self, input):
      matrix = ([[tf.cos(self.kernel[0, 0]), -tf.sin(self.kernel[0, 0])],
                [tf.sin(self.kernel[0, 0]), tf.cos(self.kernel[0, 0])]])
      return tf.matmul(input, tf.transpose(matrix))
  
  layer = Rotation2D()
  
  t = np.arange(0, 1000)/200.
  
  y_in = np.array([np.sin(t), np.cos(t)]).T
  y_ta = np.array([np.cos(t), np.sin(t)]).T
  
  model = tf.keras.Sequential()
  model.add(layer)
  
  model.compile(tf.keras.optimizers.SGD(lr=1.), loss='MSE')
  model.fit(y_in, y_ta, epochs=1)
  for i in range(100):
    print(layer.get_weights())
    model.fit(y_in, y_ta,verbose=0, batch_size=5)
  y_out = (model.predict(y_in))
  
  fig, axes = plt.subplots(2, 1)
  
  for i in range(2):
    ax = axes[i]
  
    ax.plot(y_in.T[i], label = 'input')
    ax.plot(y_ta.T[i], label = 'target')
    ax.plot(y_out.T[i], label = 'prediction')
  
  plt.legend()
  
  plt.show()```
  
  **Tags:** deep-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-20. Score: 0 | Accepted answer
  
  There are two basic problems with your code:
  
  The functions sin(t) and cos(t) (both in numpy and tensorflow) take radians as inputs. Seeing Constant(90) in your code, and the learning rate of 1. I'm guessing that you assume that it is in degrees - that's incorrect.
  
  In your training data y_ta is not a rotation of y_in:
  y_in = np.array([np.sin(t), np.cos(t)]).T
  y_ta = np.array([np.cos(t), np.sin(t)]).T
  
  
  
  It is a reflection about y=x diagonal. No wonder it fails to find an appropriate rotation.
  I just had to change y_ta to:
    y_ta = np.array([-np.cos(t), np.sin(t)]).T
  
  And train with more sensible learning rate:
    model.compile(tf.keras.optimizers.SGD(lr=0.1), loss='MSE')
    model.fit(y_in, y_ta, epochs=10)
  
  To get the angle (which I then convert to degrees):
    (180 * model.layers[0].kernel[0,0].numpy() / np.pi) % 360
    > 90.00187079311581
  
  > **Comments:**
  > **username_0**: Thanks and yes, I believe I messed up with the target value which was not actually a rotation.  
  
  ---
  
  