Hi Umang!



 Umang_Sharma:

I am trying to compute the log-sum-exp for the log_probs of these target data to compute the loss. When I use standard log-sum-exp, I get reasonable initial loss values (around 50-70) even though later it encounters some NaNs and breaks. Based on what I have read online, a numerically stable version of log-sum-exp is required to avoid this problem. However as soon I use the stable version, my loss values shoot up to the order of 15-20k. They do come down upon training but eventually they also lead to NaNs.
NOTE : I did not use the logsumexp function in PyTorch, since I needed to have a weighted summation based on my mixture components.
def log_sum_exp(self,value, weights, dim=None):
        eps = 1e-20
        m, idx = torch.max(value, dim=dim, keepdim=True)
        return m.squeeze(dim) + torch.log(torch.sum(torch.exp(value-m)*(weights.unsqueeze(2)),
                                       dim=dim) + eps)



I haven’t checked your weighted, stable log-sum-exp implementation
carefully, but, at first glance, it looks correct to me.


        m = torch.distributions.Normal(loc=mu, scale=sigma)
        probs = m.log_prob(target)

Upon adding anomaly_detection, the NaNs seem to occur at :
probs = m.log_prob(target)


If your issues really do show up first with probs = m.log_prob(target),
then your log_sum_exp() is likely not the problem.
Note that the log in log_prob() itself can overflow, depending on
the value of target and on the distribution, in particular, the value
of sigma.
Here is a brief pytorch (version 0.3.0) session that illustrates this:
>>> import torch
>>> torch.distributions.Normal (0.0, 1.0).log_prob (torch.FloatTensor ([0, 10, 1.e5, 1.e10, 1.e19, 1.e20]))

-9.1894e-01
-5.0919e+01
-5.0000e+09
-5.0000e+19
-5.0000e+37
       -inf
[torch.FloatTensor of size 6]

>>> torch.distributions.Normal (0.0, 1.e-15).log_prob (torch.FloatTensor ([0, 10, 1.e5, 1.e10, 1.e19, 1.e20]))

 3.3620e+01
-5.0000e+31
       -inf
       -inf
       -inf
       -inf
[torch.FloatTensor of size 6]

The normal distribution falls off exponentially, so it is possible have
“reasonable” values of target (and sigma) for which the normal
distribution “underflows” to 0.0, at which point the log in log_prob()
will return -inf.
I assume that target is some well-defined, static data in your training
dataset.  I would first check that your target data doesn’t contain any outlandish values.
Then I assume that your training is adjusting your value of sigma
(and mu).  If your training, for some reason, is pushing sigma
towards zero, then your log_prob() will be getting pushed into
the far tails of the normal distribution (even for reasonable target
values), and you will get -inf, as illustrated above.
So I would check for small values of sigma, perhaps with something
as simple as
    if  sigma < 1.e-10:
        print ("Yikes!  Tiny sigma!)



Seeing these huge initial loss values just by moving to the numerically stable version have led me to believe I have some bug in my current implementation.


If I understand your description correctly, it sounds like moving to your
numerically-stable log_sum_exp() postponed your training issues,
so I would say that that could be viewed as an improvement.  (I don’t
see it as evidence for a bug in log_sum_exp().)
My guess is that your training is driving your model in an unhealthy
direction (e.g., sigma --> 0).  I would check for that first.
Good luck.
K. Frank