<p>Your issue is that your function <code>pairwise_dot_sum</code>
is minimized when the dot products are negative, not when they're zero. This line:</p>
<pre><code>print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
</code></pre>
<p>is not actually what you're optimizing. The loss is approximately -1.5, (the sum of the elements of <code>dot_products.triu(diagonal=1)</code>), not 0.5.</p>
<p>The fix is to make sure that your the function is minimized when the vectors are orthogonal, for example:</p>
<pre><code>def pairwise_dot_norm(vecs):
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1)) 
    return torch.linalg.matrix_norm(dot_products.triu(diagonal=1))  # upper triangular part only   
</code></pre>
<p>p.s. There is a known, deterministic way to generate a set of N orthogonal vectors that doesn't require gradient descent: <a href="https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process</a></p>

CC BY-SA 4.0
2023-05-13T16:06:30.407Z
2023-05-13T16:06:30.407Z
bogovicj
<p>I am trying to use gradient descent to optimize 3 3-d vectors to be perpendicular. To do this I am summing the dot product of each pair of vectors. When they are all perpendicular the result should be 0.</p>
<p>This is the code I have written</p>
<pre><code>import torch

N = 3  # number of vectors
d = 3  # dimensionality of each vector

# initialize a random tensor of shape (N, d)
vecs = torch.randn(N, d, requires_grad=True)

def pairwise_dot_sum(vecs):
    &quot;&quot;&quot;
    Computes the sum of dot products of all pairs of vectors in vecs.
    &quot;&quot;&quot;
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
    return torch.sum(dot_products.triu(diagonal=1))  # upper triangular part only

optimizer = torch.optim.Adam([vecs], lr=0.01)  # choose an optimizer

with torch.no_grad():
    vecs /= torch.norm(vecs, dim=1, keepdim=True)

# train for some number of iterations
for i in range(1000):
    loss = pairwise_dot_sum(vecs)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        vecs /= torch.norm(vecs, dim=1, keepdim=True)

# check that the vectors are approximately orthogonal
dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
print(vecs)

#vecs = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])

dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(dot_products.triu(diagonal=1))

</code></pre>
<p>The output is this:</p>
<pre><code>tensor(0.5543, grad_fn=&lt;MaxBackward1&gt;)
tensor([[ 0.7315, -0.5695,  0.3749],
        [ 0.0595,  0.4699, -0.8807],
        [-0.8440,  0.0857,  0.5294]], requires_grad=True)
tensor([[ 0.0000, -0.5543, -0.4677],
        [ 0.0000,  0.0000, -0.4762],
        [ 0.0000,  0.0000,  0.0000]], grad_fn=&lt;TriuBackward0&gt;)
</code></pre>
<p>So you can see that the loss is 0.5, instead of 0.</p>
<p>and the vectors seem to form an equilateral triangle centered around the origin.</p>
<p>What mistake have I made? How can I get this fix this to correctly find orthogonal vectors?</p>

CC BY-SA 4.0
2023-05-13T19:50:17.720Z
There's no need to reinvent the wheel -- PyTorch allows you to implement orthogonal constraints on parameters. See: https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html
Sycorax
<machine-learning><training>
gradient descent optimization to make orthogonal vectors
river
# gradient descent optimization to make orthogonal vectors

(Asked by: username_0 on 2023-05-01)

I am trying to use gradient descent to optimize 3 3-d vectors to be perpendicular. To do this I am summing the dot product of each pair of vectors. When they are all perpendicular the result should be 0.
This is the code I have written
import torch

N = 3  # number of vectors
d = 3  # dimensionality of each vector

# initialize a random tensor of shape (N, d)
vecs = torch.randn(N, d, requires_grad=True)

def pairwise_dot_sum(vecs):
    """
    Computes the sum of dot products of all pairs of vectors in vecs.
    """
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
    return torch.sum(dot_products.triu(diagonal=1))  # upper triangular part only

optimizer = torch.optim.Adam([vecs], lr=0.01)  # choose an optimizer

with torch.no_grad():
    vecs /= torch.norm(vecs, dim=1, keepdim=True)

# train for some number of iterations
for i in range(1000):
    loss = pairwise_dot_sum(vecs)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    with torch.no_grad():
        vecs /= torch.norm(vecs, dim=1, keepdim=True)

# check that the vectors are approximately orthogonal
dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(torch.max(torch.abs(dot_products - torch.eye(N, N))))
print(vecs)

#vecs = torch.tensor([[1,0,0],[0,1,0],[0,0,1]])

dot_products = torch.matmul(vecs, vecs.transpose(0, 1))
print(dot_products.triu(diagonal=1))


The output is this:
tensor(0.5543, grad_fn=)
tensor([[ 0.7315, -0.5695,  0.3749],
        [ 0.0595,  0.4699, -0.8807],
        [-0.8440,  0.0857,  0.5294]], requires_grad=True)
tensor([[ 0.0000, -0.5543, -0.4677],
        [ 0.0000,  0.0000, -0.4762],
        [ 0.0000,  0.0000,  0.0000]], grad_fn=)

So you can see that the loss is 0.5, instead of 0.
and the vectors seem to form an equilateral triangle centered around the origin.
What mistake have I made? How can I get this fix this to correctly find orthogonal vectors?

**Tags:** machine-learning, training

> **Comments:**
> **username_1**: There's no need to reinvent the wheel -- PyTorch allows you to implement orthogonal constraints on parameters. See: https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.orthogonal.html  

---

## 1 answers

### Answer by username_2 on 2023-05-13. Score: 2 | Accepted answer

Your issue is that your function pairwise_dot_sum
is minimized when the dot products are negative, not when they're zero. This line:
print(torch.max(torch.abs(dot_products - torch.eye(N, N))))

is not actually what you're optimizing. The loss is approximately -1.5, (the sum of the elements of dot_products.triu(diagonal=1)), not 0.5.
The fix is to make sure that your the function is minimized when the vectors are orthogonal, for example:
def pairwise_dot_norm(vecs):
    dot_products = torch.matmul(vecs, vecs.transpose(0, 1)) 
    return torch.linalg.matrix_norm(dot_products.triu(diagonal=1))  # upper triangular part only   

p.s. There is a known, deterministic way to generate a set of N orthogonal vectors that doesn't require gradient descent: https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process


---

