@tom Hi Thomas and hope you are doing alright.
I have a couple of questions in regard to the porting of code.

there is an optimizer that I have ported and I would like to have your opinion.

the Theano/Lasagne version:
def geoSGD(loss_or_grads, params, learning_rate):
“”“Geodesic Stochastic Gradient Descent (geoSGD) updates
Generates update expressions of the form:
* param := param - learning_rate * gradient
Parameters
----------
loss_or_grads : symbolic expression or list of expressions
A scalar loss expression, or a list of gradient expressions
params : list of shared variables
The variables to generate update expressions for (in our case they are: hh_W_u and hh_W_v)
learning_rate : float or symbolic scalar
The learning rate controlling the size of update steps
Returns
-------
OrderedDict
A dictionary mapping each parameter to its update expression
“””
grads = get_or_compute_grads(loss_or_grads, params)
updates = OrderedDict()
lr = learning_rate
for param, grad in zip(params, grads):
    W = param.get_value(borrow=True)
    G = grad
    A = T.dot(G, W.T) - T.dot(W, G.T)  # A = G * M.T - M * G.T
    I = T.identity_like(A)
    cayley = T.dot(T.nlinalg.matrix_inverse(I+(lr/2.)*A), I-(lr/2.)*A)  # (I + eta/2 * A)**(-1) - (I - eta/2 * A)
    updates[param] = T.dot(cayley, W)  # cayley * M = ((I + eta/2 * A)**(-1) - (I - eta/2 * A)) * M
return updates

my PyTorch version:
def geoSGD(outputs, params, learning_rate):
grads_ = torch.autograd.grad(outputs, params, retain_graph=True, allow_unused=True) # had to add ‘retain_graph=True’
updates = OrderedDict()
lr = learning_rate

for param, grad_ in zip(params, grads_):
    W = param.double()
    G = grad_.double()
    A = torch.mm(G, W.transpose(0, 1)) - torch.mm(W, G.transpose(0, 1))

    if torch.all(torch.eq(A.transpose(0, 1), -A)):
        if torch.sum(abs(A.transpose(0,  1) + A)) == 0:
            print('The matrix A is skew symmetric')
    else:
        print('The matrix A is NOT skew symmetric')
    
    I = torch.eye(A.size()[0],A.size()[1]).double()
    cayley = torch.mm(torch.inverse(I+(lr/2.)*A), I-(lr/2.)*A)  # (I + eta/2 * A)**(-1) - (I - eta/2 * A)
    updates[param] = torch.mm(cayley, W)  # cayley * M = ((I + eta/2 * A)**(-1) - (I - eta/2 * A)) * M
return updates

what I do not understand fully is that the ‘loss_or_grads’ have to be replaced by ‘outputs’ and, always if my porting is correct, how does the grad() function in PyTorch version knows the ‘loss’ to do the calculations and what ‘output’ is exactly in the case of PyTorch.

In this code among other things I have to deal with factorization of parameters and their gradients. Maybe I am wrong but for example if I have a situation like this:
W = U * S * V
F(W) = A * W + B
dF(W)/dU is not possible

only if I do like following
F(U, S, V) = A * (U * S * V) + B
then
dF(U, S, V)/dU is possible
Is there any way to have simultaneously
dF/dW, dF/dU, dF/dS, dF/dV               (‘d’ all the time means partial derivative)
Thank you very much in advance for your time, help and understanding.
Cheers.
Ergnoor