What is the statistical model for a multi-label problem? In a setting with a binary $y$ like dog/cat, a reasonable statistical model is to posit that the probability parameter $p$ of a $\text{Binomial}(1, 0)$ distribution is some function $f$ of features $X$. This leads to many common machine learning approaches like logistic regression and neural networks.
In a setting with multiple classes in $y$, such as dog/cat/horse, a reasonable statistical model is that the multiple probability values $\vec p$ of a $\text{Multinomial}(1, \vec p)$ distribution is some function of features $X$. Much like in the binary setting, this leads to many common approaches like multinomial logistic regression and various forms of deep learning (e.g., convolutional neural networks for MNIST handwritten digits).
In a multi-label setting, such as identifying if a photograph contains a dog, a cat, or both, what would be the statistical model?
(To a large extent, I think I mean the formal likelihood, regardless of the functional form of expressing the probability of class membership as depending on the features, but I want to leave it a bit vague to allow for an answer that I’m thinking about it wrong by thinking of statistical likelihood.)
EDIT
The comments have clarified that an Ising likelihood works. With that being the case, how do the prior probabilities of the classes come into the picture? For instance, in a logistic regression, if I have $99$ $0$s for every $1$, I expect low probabilities of $1$ unless the features are extremely informative. In a multi-label setting, it seems like the prior probability of each class would be the ratio of that class to all possible alternatives, which I would consider to be zero: out of every possible sight there is to see, the probability of seeing a dog ought to be tiny or even zero (and the fact that we’re on Earth (for now) and near dogs is what allows us to see dogs with frequency).
It seems like something like this fails for multi-label classification.

You can think of the multinomial (aka multinoulli) distribution used in multinomial classification as a multivariate discrete distribution obtained in two steps:

*

*assign a Bernoulli marginal distribution to each of the labels;

*use a very specific copula function to model the dependence between the marginals; the copula is such that the Bernoullis are perfectly dependent (if one of them takes value 1, all the others take value 0).

In a multi-label problem, you change the copula because you drop the assumption of perfect dependence (two or more Bernoullis are allowed to simultaneously take value 1).
The problem is that you need to decide which copula to use. For example, you could use the product copula (all the Bernoullis are independent), similarly to what is done in naive Bayes models.
There are several obstacles to choosing / specifying an appropriate copula function:

*

*if the number of labels ($K$) is large, you incur in the curse of dimensionality; roughly speaking, you need to model $K^2$ correlations between the Bernoullis, as mentioned in Ben's answer;

*there are infinitely many possible dependence structures you could choose;

*it's unclear whether the dependence structure needs to be parametrized separately (before training the model) or estimated while training the model;

*the copula of a discrete random vector is not fully identifiable, which can cause serious inconsistencies.

That said, I think that the easier way to go is to use the product copula (make the Bernoullis independent), which is equivalent to training $K$ separate binomial classification models (one for each label), but with parameter sharing in all the layers but the last one.
