parametrized function fθ without enforcing(cid:82) fθ(x)dx = 1 during training

• let’s ﬁrst introduce a noise variable (cid:15) ∼ p(cid:15) and let’s consider the following mixture distribution

T ∼ B(s), s ∈ (0, 1)

V ∼ X1{T =1} + ε1{T =0}

• using Bayes rule and denoting η = (1 − s)/s we have

pT|V (T = 1|V = v) =

pV |T (V = v|T = 1)

pV |T (V = v|T = 1) + ηpV |T (V = v|T = 0)

,

• parametrize pV |T (V = v|T = 1) = fθ(v) exp(c) with fθ > 0 and learnable parameters {θ, c}
• minimize the NLL of logistic regression (usual binary classiﬁcation set-up)
L(θ, c) = −E(v,t)∼(V,T ) log[pT|V (T = t|V = v)]

• the minimum is attained at fθ∗ exp(c∗) = pX if pX (v) = 0 =⇒ p(cid:15)(v) > 0. If fθ is powerful enough, one

can set c = 0 and the model will self-normalize [Mnih and Teh, 2012]
A Cookbook of Self-Supervised Learning