<p>as far as I understand it, it's not the log-probability but the log-likelihood.
the likelihood can be greater than 1 so negative loss is possible.</p>

CC BY-SA 4.0
2022-09-19T11:08:30.053Z
2022-09-19T11:08:30.053Z
His Royal Dudeness
<p>I don't know if this comment will be helpful, but shouldn't the sum of the log determinant of the Jacobian (LDJ) have opposite sign in the forward and inverse pass? I'm not talking about the LDJ being sum of the positive scaling function in the forward and sum of the negative of the scaling function in the inverse, I'm talking about the LDJ itself. </p>

<p>For example, from the change of variable formula, 
<span class="math-container">$$p_z(z) \ dz = p_x(x) \ dx$$</span>
differentiating w.r.t <span class="math-container">$\ dz$</span> gives,
<span class="math-container">$$ p_z(z) = p_x(x) \left|\frac{dx}{dz}\right|$$</span>
taking the log of both sides gives,
<span class="math-container">$$ ln(p_z(z)) = ln(p_x(x)) +  ln\left(\left|\frac{dx}{dz}\right|\right)$$</span>
Now, if we repeat this process but differentitating w.r.t <span class="math-container">$\ dx$</span>, 
<span class="math-container">$$ p_x(x) = p_z(z) \left|\frac{dz}{dx}\right|$$</span>
taking the log and rearranging for <span class="math-container">$ln(p_z(z))$</span> gives,
taking the log of both sides gives,
<span class="math-container">$$ ln(p_z(z)) = ln(p_x(x)) -  ln\left(\left|\frac{dz}{dx}\right|\right)$$</span></p>

<p>A change of sign occurs when the Jacobian is reversed. Could this be your problem? Also, as @chris-cundy said you can have a probability density greater than 1 at a single point. Remember it is the integral over all space that cannot be greater than one, an individual point between greater than 1 is fine.</p>

CC BY-SA 4.0
2020-05-13T10:59:13.767Z
2020-05-13T10:59:13.767Z
AlphaBetaGamma96
<p>I am trying to use RealNVP with some data I have (the input size is a 1D vector of size 22). <a href="https://arxiv.org/abs/1605.08803" rel="nofollow noreferrer">Here</a> is the link to the RealNVP paper and <a href="https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html" rel="nofollow noreferrer">here</a> is a nice, short explanation of it (the paper is pretty long). My code is mainly based on <a href="https://github.com/tonyduan/normalizing-flows" rel="nofollow noreferrer">this</a> code from GitHub and below are the main piece that I am using (with slight adjustments). The problem is that the loss is getting negative, which in the definition of my code means that the log-probability of the my data is positive, which in turn means that the probabilities are bigger than 1. This is impossible mathematically, and I see no way how this can happen, from a mathematical point of view. I also couldn't find a mistake in my code. Can someone help me with this? Is there a mistake there? Am I missing something with my understanding of normalizing flows? Thank you! </p>

<pre><code>class NormalizingFlowModel(nn.Module):

    def __init__(self, prior, flows):
        super().__init__()
        self.prior = prior
        self.flows = nn.ModuleList(flows)

    def forward(self, x):
        m, _ = x.shape
        log_det = torch.zeros(m).cuda()
        for flow in self.flows:
            x, ld = flow.forward(x)
            log_det += ld
        z, prior_logprob = x, self.prior.log_prob(x)
        return z, prior_logprob, log_det

    def inverse(self, z):
        m, _ = z.shape
        log_det = torch.zeros(m).cuda()
        for flow in self.flows[::-1]:
            z, ld = flow.inverse(z)
            log_det += ld
        x = z
        return x, log_det

    def sample(self, n_samples):
        z = self.prior.sample((n_samples,))
        x, _ = self.inverse(z)
        return x


class FCNN_for_NVP(nn.Module):
    """
    Simple fully connected neural network to be used for Real NVP
    """
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(in_dim, 32),
            nn.Tanh(),
            nn.Linear(32, 32),
            nn.Tanh(),
            nn.Linear(32, 64),
            nn.Tanh(),
            nn.Linear(64, 64),
            nn.Tanh(),
            nn.Linear(64, 32),
            nn.Tanh(),
            nn.Linear(32, 32),
            nn.Tanh(),
            nn.Linear(32, out_dim),
        )

    def forward(self, x):
        return self.network(x)


class RealNVP(nn.Module):
    """
    Non-volume preserving flow.

    [Dinh et. al. 2017]
    """
    def __init__(self, dim, base_network=FCNN_for_NVP):
        super().__init__()
        self.dim = dim
        self.t1 = base_network(dim // 2, dim // 2)
        self.s1 = base_network(dim // 2, dim // 2)
        self.t2 = base_network(dim // 2, dim // 2)
        self.s2 = base_network(dim // 2, dim // 2)

    def forward(self, x):
        lower, upper = x[:,:self.dim // 2], x[:,self.dim // 2:]      
        t1_transformed = self.t1(lower)
        s1_transformed = self.s1(lower)
        upper = t1_transformed + upper * torch.exp(s1_transformed)
        t2_transformed = self.t2(upper)
        s2_transformed = self.s2(upper)
        lower = t2_transformed + lower * torch.exp(s2_transformed)
        z = torch.cat([lower, upper], dim=1)
        log_det = torch.sum(s1_transformed, dim=1) + torch.sum(s2_transformed, dim=1)
        return z, log_det

    def inverse(self, z):
        lower, upper = z[:,:self.dim // 2], z[:,self.dim // 2:]
        t2_transformed = self.t2(upper)
        s2_transformed = self.s2(upper)
        lower = (lower - t2_transformed) * torch.exp(-s2_transformed)
        t1_transformed = self.t1(lower)
        s1_transformed = self.s1(lower)
        upper = (upper - t1_transformed) * torch.exp(-s1_transformed)
        x = torch.cat([lower, upper], dim=1)
        log_det = torch.sum(-s1_transformed, dim=1) + torch.sum(-s2_transformed, dim=1)
        return x, log_det

flow = RealNVP(dim=data.size(1))
flows = [flow for _ in range(1)]
prior = MultivariateNormal(torch.zeros(data.size(1)).cuda(), torch.eye(data.size(1)).cuda())
model = NormalizingFlowModel(prior, flows)
model = model.cuda()

for i in range(10):
    for j, dtt in enumerate(my_dataloader_bkg_only):
        optimizer.zero_grad()
        x = dtt[0].float()
        z, prior_logprob, log_det = model(x)
        logprob = prior_logprob + log_det
        loss = -torch.mean(prior_logprob + log_det)
        loss.backward()
        optimizer.step()
    if i % 1 == 0:
        print("Saved")
        best_loss = logprob.mean().data.cpu().numpy()
        print(logprob.mean().data.cpu().numpy(), prior_logprob.mean().data.cpu().numpy(),
                  log_det.mean().data.cpu().numpy())
</code></pre>

CC BY-SA 4.0
2020-02-02T02:41:25.657Z
You might well have a bug, but in general there's no reason to not have a log-likelihood greater than 0. E.g. consider fitting a single 1-d point with a Gaussian with parameterised mean and variance. The maximum-likelihood solution is an infinitely sharp peak with infinite log likelihood.
Chris Cundy
CC BY-SA 4.0
2020-02-02T03:19:23.370Z
So how should I think about a probability bigger than 1? Assuming I have 2 1-d points (in my case I have 1M), and one of them has a probability of, say, 1.5, how can I make sense of that mathematically? Thank you!
JohnDoe122
CC BY-SA 4.0
2020-02-02T03:54:16.897Z
You are computing a probability density (flow models are density models), so you should interpret it as a probability density. Maybe this will help https://towardsdatascience.com/probability-concepts-explained-probability-distributions-introduction-part-3-4a5db81858dc
Chris Cundy
<objective-functions><pytorch>
RealNVP gives wrong probabilities
JohnDoe122
# RealNVP gives wrong probabilities

(Asked by: username_0 on 2020-02-02)

I am trying to use RealNVP with some data I have (the input size is a 1D vector of size 22). Here is the link to the RealNVP paper and here is a nice, short explanation of it (the paper is pretty long). My code is mainly based on this code from GitHub and below are the main piece that I am using (with slight adjustments). The problem is that the loss is getting negative, which in the definition of my code means that the log-probability of the my data is positive, which in turn means that the probabilities are bigger than 1. This is impossible mathematically, and I see no way how this can happen, from a mathematical point of view. I also couldn't find a mistake in my code. Can someone help me with this? Is there a mistake there? Am I missing something with my understanding of normalizing flows? Thank you! 
class NormalizingFlowModel(nn.Module):

    def __init__(self, prior, flows):
        super().__init__()
        self.prior = prior
        self.flows = nn.ModuleList(flows)

    def forward(self, x):
        m, _ = x.shape
        log_det = torch.zeros(m).cuda()
        for flow in self.flows:
            x, ld = flow.forward(x)
            log_det += ld
        z, prior_logprob = x, self.prior.log_prob(x)
        return z, prior_logprob, log_det

    def inverse(self, z):
        m, _ = z.shape
        log_det = torch.zeros(m).cuda()
        for flow in self.flows[::-1]:
            z, ld = flow.inverse(z)
            log_det += ld
        x = z
        return x, log_det

    def sample(self, n_samples):
        z = self.prior.sample((n_samples,))
        x, _ = self.inverse(z)
        return x


class FCNN_for_NVP(nn.Module):
    """
    Simple fully connected neural network to be used for Real NVP
    """
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(in_dim, 32),
            nn.Tanh(),
            nn.Linear(32, 32),
            nn.Tanh(),
            nn.Linear(32, 64),
            nn.Tanh(),
            nn.Linear(64, 64),
            nn.Tanh(),
            nn.Linear(64, 32),
            nn.Tanh(),
            nn.Linear(32, 32),
            nn.Tanh(),
            nn.Linear(32, out_dim),
        )

    def forward(self, x):
        return self.network(x)


class RealNVP(nn.Module):
    """
    Non-volume preserving flow.

    [Dinh et. al. 2017]
    """
    def __init__(self, dim, base_network=FCNN_for_NVP):
        super().__init__()
        self.dim = dim
        self.t1 = base_network(dim // 2, dim // 2)
        self.s1 = base_network(dim // 2, dim // 2)
        self.t2 = base_network(dim // 2, dim // 2)
        self.s2 = base_network(dim // 2, dim // 2)

    def forward(self, x):
        lower, upper = x[:,:self.dim // 2], x[:,self.dim // 2:]      
        t1_transformed = self.t1(lower)
        s1_transformed = self.s1(lower)
        upper = t1_transformed + upper * torch.exp(s1_transformed)
        t2_transformed = self.t2(upper)
        s2_transformed = self.s2(upper)
        lower = t2_transformed + lower * torch.exp(s2_transformed)
        z = torch.cat([lower, upper], dim=1)
        log_det = torch.sum(s1_transformed, dim=1) + torch.sum(s2_transformed, dim=1)
        return z, log_det

    def inverse(self, z):
        lower, upper = z[:,:self.dim // 2], z[:,self.dim // 2:]
        t2_transformed = self.t2(upper)
        s2_transformed = self.s2(upper)
        lower = (lower - t2_transformed) * torch.exp(-s2_transformed)
        t1_transformed = self.t1(lower)
        s1_transformed = self.s1(lower)
        upper = (upper - t1_transformed) * torch.exp(-s1_transformed)
        x = torch.cat([lower, upper], dim=1)
        log_det = torch.sum(-s1_transformed, dim=1) + torch.sum(-s2_transformed, dim=1)
        return x, log_det

flow = RealNVP(dim=data.size(1))
flows = [flow for _ in range(1)]
prior = MultivariateNormal(torch.zeros(data.size(1)).cuda(), torch.eye(data.size(1)).cuda())
model = NormalizingFlowModel(prior, flows)
model = model.cuda()

for i in range(10):
    for j, dtt in enumerate(my_dataloader_bkg_only):
        optimizer.zero_grad()
        x = dtt[0].float()
        z, prior_logprob, log_det = model(x)
        logprob = prior_logprob + log_det
        loss = -torch.mean(prior_logprob + log_det)
        loss.backward()
        optimizer.step()
    if i % 1 == 0:
        print("Saved")
        best_loss = logprob.mean().data.cpu().numpy()
        print(logprob.mean().data.cpu().numpy(), prior_logprob.mean().data.cpu().numpy(),
                  log_det.mean().data.cpu().numpy())

**Tags:** objective-functions, pytorch

> **Comments:**
> **username_1**: You might well have a bug, but in general there's no reason to not have a log-likelihood greater than 0. E.g. consider fitting a single 1-d point with a Gaussian with parameterised mean and variance. The maximum-likelihood solution is an infinitely sharp peak with infinite log likelihood.  
> **username_0**: So how should I think about a probability bigger than 1? Assuming I have 2 1-d points (in my case I have 1M), and one of them has a probability of, say, 1.5, how can I make sense of that mathematically? Thank you!  
> **username_1**: You are computing a probability density (flow models are density models), so you should interpret it as a probability density. Maybe this will help https://towardsdatascience.com/probability-concepts-explained-probability-distributions-introduction-part-3-4a5db81858dc  

---

## 2 answers

### Answer by username_2 on 2022-09-19. Score: 0

as far as I understand it, it's not the log-probability but the log-likelihood.
the likelihood can be greater than 1 so negative loss is possible.


---

### Answer by username_3 on 2020-05-13. Score: 0

I don't know if this comment will be helpful, but shouldn't the sum of the log determinant of the Jacobian (LDJ) have opposite sign in the forward and inverse pass? I'm not talking about the LDJ being sum of the positive scaling function in the forward and sum of the negative of the scaling function in the inverse, I'm talking about the LDJ itself.

For example, from the change of variable formula,
$$p\_z(z) \ dz = p\_x(x) \ dx$$
differentiating w.r.t $\ dz$ gives,
$$ p\_z(z) = p\_x(x) \left|\frac{dx}{dz}\right|$$
taking the log of both sides gives,
$$ ln(p\_z(z)) = ln(p\_x(x)) + ln\left(\left|\frac{dx}{dz}\right|\right)$$
Now, if we repeat this process but differentitating w.r.t $\ dx$,
$$ p\_x(x) = p\_z(z) \left|\frac{dz}{dx}\right|$$
taking the log and rearranging for $ln(p\_z(z))$ gives,
taking the log of both sides gives,
$$ ln(p\_z(z)) = ln(p\_x(x)) - ln\left(\left|\frac{dz}{dx}\right|\right)$$

A change of sign occurs when the Jacobian is reversed. Could this be your problem? Also, as @chris-cundy said you can have a probability density greater than 1 at a single point. Remember it is the integral over all space that cannot be greater than one, an individual point between greater than 1 is fine.


---

