Binary classification but learning the true probability I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would like to learn this function the best way I can.

*

*Would that be correct to say that logistic regression is actually focused on approximating $p$ with some predictor $q$ which minimizes the cross-entropy loss $\mathrm{CE}(p,q)$?


*What are the other useful methods to approximate $p$ based off the data $D$? Of course I can replace a linear model in logistic regression with some non-linear counterpart $q_\theta:\Bbb R^d \to [0,1]$ and look for $\theta$ that minimizes $\mathrm CE(p,q_\theta)$, searching for example over NNs or random forests or whatever other non-linear point estimators come to mind. Perhaps, some Bayesian frameworks would come handy here?


*If I need to learn the entropy of $p(x)$ at a given $x$ of course I can just approximate it with the entropy of $q(x)$: if $p\approx q$ then we should expect $H(p) \approx H(q)$. I wonder however whether there is a direct way to learn $H(p)$ as a function of $x$, perhaps with me wanting to minimize $\mathrm{MSE}(H(p), f_\theta)$ where $f_\theta$ is some parametrized positive-valued function.

*

*Yes, this is true. $q$ is chosen here from a specific family of functions, parametrized by the intercept and slope of the logistic regression.

*Any binary probabilistic classification model would work well. The best method would depend on the nature of $p$, so the more methods you try, the better it would be. Usually, one of the following methods works well enough: logistic regression / neural networks (multilayer perceptron, in your case) / gradient boosting with small decision trees / KNN.  Bayesian methods can work with any of these frameworks as a way of regularizing (= assigning priors to) them.

*I don't know about methods of direct evaluation of entropy, and I think that it is perfectly OK to evaluate it using the entropy of q, given that q itself is well calibrated.

