Hi Seyhmus!



 niceonede:

In the main program I used this one:
    loss = my_loss.apply(a[0],b[0],m[0],s[0],a2,b2,m2,s2)

Here the loss is of this sort:
    tensor([1.1734])

The rest of the code is the same as I had written in the original question. I got an error as this one:
    element 0 of tensors does not require grad and does not have a grad_fn



These three pieces of information strongly suggest that none of
the arguments you are passing into my_loss.apply() have
requires_grad = True.  Try printing out the arguments right
before calling my_loss.apply() to see whether they show up
with requires_grad = True.
Looking at your code – and making some assumptions to fill in the
gaps – a, b, etc., come from parameter1, parameter2, etc., so I
have no reason to think that they have requires_grad = True.
Furthermore, even though a2, b2, etc., come from output, which,
as the result of model, presumably has requires_grad = True,
you then call .item(), which returns a python scalar (rather than a
tensor), and, as such, strips away any requires_grad property.
If you are trying to optimize the parameters of the model that
produces a2, b2, etc., then you have to move these calls to .item().
inside of my_loss.forward().  Pass a2, etc., into my_loss.apply()  as
python tensors (and do check that they have requires_grad = True
to make sure that something else isn’t getting broken) and convert them
to python scalars (in order to pass them to quad()) with calls to .item()
inside of forward().


a quick fix would be to use Variable after the definition of loss.


As a general rule, if you find yourself tempted to “manually” turn on
requires_grad = True somewhere in the middle of a calculation,
you should resist that temptation.  Such a “quick fix” is almost never
an actual fix.  Typically, you have “broken the computation graph”
somewhere upstream of your purported fix, and you need find and
fix the actual, upstream problem.  (As noted above, in your case, the
calls to .item() are likely the culprit that has broken the computation
graph.)
As an aside, Variable has been deprecated for some time now, and
the autograd functionality in Variable has since been subsumed into
pytorch tensors, so just use regular tensors.


    @staticmethod
    def backward(ctx, grad_output):
        a,b,m,s,a2,b2,m2,s2 = ctx.saved_variables
        grad_alpha = quad(integrand_alpha, 0, 5*10**-7, args=(a,b,m,s,a2,b2,m2,s2), points=(max(0,min(m-2*s,m2-2*s2)), max(m+2*s,m2+2*s2)))[0]
        grad_beta = quad(integrand_beta, 0, 5*10**-7, args=(a,b,m,s,a2,b2,m2,s2), points=(max(0,min(m-2*s,m2-2*s2)), max(m+2*s,m2+2*s2)))[0]
        grad_mu = quad(integrand_mu, 0, 5*10**-7, args=(a,b,m,s,a2,b2,m2,s2), points=(max(0,min(m-2*s,m2-2*s2)), max(m+2*s,m2+2*s2)))[0]
        grad_sig = quad(integrand_sigma, 0, 5*10**-7, args=(a,b,m,s,a2,b2,m2,s2), points=(max(0,min(m-2*s,m2-2*s2)), max(m+2*s,m2+2*s2)))[0]
                
        return torch.FloatTensor(grad_alpha), torch.FloatTensor(grad_beta), torch.FloatTensor(grad_mu), torch.FloatTensor(grad_sig), None



Your version of backward() has a flaw.  For reasons I won’t go into,
it should work in the particular context in which you call it, but it’s
wrong because its output does not depend on its grad_output
argument.
Let’s assume for simplicity that you write a Function whose forward()
method takes a single tensor as an argument and returns a single
tensor.  That is, it takes a collection of scalar arguments and returns a
collection of scalar results.  The derivative of such a function is the
so-called Jacobian, that is, the matrix of partial derivatives of all of
the results with respect to all of the arguments.
Your backward() function – which we think of as calculating the
derivative of the corresponding forward() –  doesn’t return the full
Jacobian (although it might choose to compute the full Jacobian
internally).  Rather it returns the matrix-vector product of the Jacobian
matrix with the grad_output vector that pytorch’s autograd framework
passes into your backward() function.
For your my_loss Function to work correctly in (perfectly plausible)
more general contexts, its backward() method can’t simply ignore
its grad_output argument, but instead has to use it to compute (and
return) the expected Jacobian-grad_output product.
(The Jacobian-grad_output product returned by your backward()
will then be passed in as the grad_output argument of the backward()
of the Function next upstream in the computation graph.  That’s how
autograd implements the chain rule.)
Best.
K. Frank