Answers.Body: <p>Your issue is that your function <code>pairwise_dot_sum</code>
  is minimized when the dot products are negative, not when they're zero. This line:</p>
  <pre><code>print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
  </code></pre>
  <p>is not actually what you're optimizing. The loss is approximately -1.5, (the sum of the elements of <code>dot_products.triu(diagonal=1)</code>), not 0.5.</p>
  <p>The fix is to make sure that your the function is minimized when the vectors are orthogonal, for example:</p>
  <pre><code>def pairwise_dot_norm(vecs):
      dot_products = torch.matmul(vecs, vecs.transpose(0, 1)) 
      return torch.linalg.matrix_norm(dot_products.triu(diagonal=1))  # upper triangular part only   
  </code></pre>
  <p>p.s. There is a known, deterministic way to generate a set of N orthogonal vectors that doesn't require gradient descent: <a href="https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-13T16:06:30.407Z
Answers.LastActivityDate: 2023-05-13T16:06:30.407Z
Answers.UserDisplayName: bogovicj
Body: <p>I am trying to use gradient descent to optimize 3 3-d vectors to be perpendicular. To do this I am summing the dot product of each pair of vectors. When they are all perpendicular the result should be 0.</p>
  <p>This is the code I have written</p>
  <pre><code>import torch
  
  N = 3  # number of vectors
  d = 3  # dimensionality of each vector
  
  # initialize a random tensor of shape (N, d)
  vecs = torch.randn(N, d, requires_grad=True)
  
  def pairwise_dot_sum(vecs):
      &quot;&quot;&quot;
      Computes the sum of dot products of all pairs of vectors in vecs.
      &quot;&quot;&quot;
      dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
      return torch.sum(dot_products.triu(diagonal=1))  # upper triangular part only
  
  optimizer = torch.optim.Adam([vecs], lr=0.01)  # choose an optimizer
  
  with torch.no_grad():
      vecs /= torch.norm(vecs, dim=1, keepdim=True)
  
  # train for some number of iterations
  for i in range(1000):
      loss = pairwise_dot_sum(vecs)
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      with torch.no_grad():
          vecs /= torch.norm(vecs, dim=1, keepdim=True)
  
  # check that the vectors are approximately orthogonal
  dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
  print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
  print(vecs)
  
  #vecs = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])
  
  dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
  print(dot_products.triu(diagonal=1))
  
  </code></pre>
  <p>The output is this:</p>
  <pre><code>tensor(0.5543, grad_fn=&lt;MaxBackward1&gt;)
  tensor([[ 0.7315, -0.5695,  0.3749],
          [ 0.0595,  0.4699, -0.8807],
          [-0.8440,  0.0857,  0.5294]], requires_grad=True)
  tensor([[ 0.0000, -0.5543, -0.4677],
          [ 0.0000,  0.0000, -0.4762],
          [ 0.0000,  0.0000,  0.0000]], grad_fn=&lt;TriuBackward0&gt;)
  </code></pre>
  <p>So you can see that the loss is 0.5, instead of 0.</p>
  <p>and the vectors seem to form an equilateral triangle centered around the origin.</p>
  <p>What mistake have I made? How can I get this fix this to correctly find orthogonal vectors?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-13T19:50:17.720Z
Comments.Text: There's no need to reinvent the wheel -- PyTorch allows you to implement orthogonal constraints on parameters. See: https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html
Comments.UserDisplayName: Sycorax
Tags: <machine-learning><training>
Title: gradient descent optimization to make orthogonal vectors
UserDisplayName: river
ThreadText: # gradient descent optimization to make orthogonal vectors
  
  (Asked by: username_0 on 2023-05-01)
  
  I am trying to use gradient descent to optimize 3 3-d vectors to be perpendicular. To do this I am summing the dot product of each pair of vectors. When they are all perpendicular the result should be 0.
  
  This is the code I have written
  
  ```
  import torch
  
  N = 3  # number of vectors
  d = 3  # dimensionality of each vector
  
  # initialize a random tensor of shape (N, d)
  vecs = torch.randn(N, d, requires_grad=True)
  
  def pairwise_dot_sum(vecs):
      """
      Computes the sum of dot products of all pairs of vectors in vecs.
      """
      dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
      return torch.sum(dot_products.triu(diagonal=1))  # upper triangular part only
  
  optimizer = torch.optim.Adam([vecs], lr=0.01)  # choose an optimizer
  
  with torch.no_grad():
      vecs /= torch.norm(vecs, dim=1, keepdim=True)
  
  # train for some number of iterations
  for i in range(1000):
      loss = pairwise_dot_sum(vecs)
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      with torch.no_grad():
          vecs /= torch.norm(vecs, dim=1, keepdim=True)
  
  # check that the vectors are approximately orthogonal
  dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
  print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
  print(vecs)
  
  #vecs = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])
  
  dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
  print(dot_products.triu(diagonal=1))
  
  ```
  
  The output is this:
  
  ```
  tensor(0.5543, grad_fn=<MaxBackward1>)
  tensor([[ 0.7315, -0.5695,  0.3749],
          [ 0.0595,  0.4699, -0.8807],
          [-0.8440,  0.0857,  0.5294]], requires_grad=True)
  tensor([[ 0.0000, -0.5543, -0.4677],
          [ 0.0000,  0.0000, -0.4762],
          [ 0.0000,  0.0000,  0.0000]], grad_fn=<TriuBackward0>)
  
  ```
  
  So you can see that the loss is 0.5, instead of 0.
  
  and the vectors seem to form an equilateral triangle centered around the origin.
  
  What mistake have I made? How can I get this fix this to correctly find orthogonal vectors?
  
  **Tags:** machine-learning, training
  
  > **Comments:**
  > **username_1**: There's no need to reinvent the wheel -- PyTorch allows you to implement orthogonal constraints on parameters. See: https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-13. Score: 2 | Accepted answer
  
  Your issue is that your function `pairwise_dot_sum`
  is minimized when the dot products are negative, not when they're zero. This line:
  
  ```
  print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
  
  ```
  
  is not actually what you're optimizing. The loss is approximately -1.5, (the sum of the elements of `dot_products.triu(diagonal=1)`), not 0.5.
  
  The fix is to make sure that your the function is minimized when the vectors are orthogonal, for example:
  
  ```
  def pairwise_dot_norm(vecs):
      dot_products = torch.matmul(vecs, vecs.transpose(0, 1)) 
      return torch.linalg.matrix_norm(dot_products.triu(diagonal=1))  # upper triangular part only   
  
  ```
  
  p.s. There is a known, deterministic way to generate a set of N orthogonal vectors that doesn't require gradient descent: <https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process>
  
  
  ---
  
  