Logistic regression cost function finding multiple minima in matlab using fminunc Good afternoon,
I am a beginner at machine learning and am trying to implement a basic logistic regression using the formula for the cost function we learned in class:
https://miro.medium.com/max/2836/1*_52kKSp8zWgVTNtnE2eYrg.png (cost function)
https://miro.medium.com/max/2964/1*o4Dy1w4n2kDOLA8UEwGC9g.png (hypothesis function)
I implemented the following code, and it looks like depending on my initial guess, fminunc usually finds a local minimum near to the guess, but also sometimes stops for other reasons (such as "undefined at initial point" or "cannot decrease the objective function along the search direction") indicating there are many local minima in addition to potentially other issues.  I also have similar issues with fminsearch, and updated the code below with fminsearch.  If anyone with more experience could provide some insight, it would be greatly appreciated.
Here is my code:
%logistic regression: predicting benign vs. malignant tumors with data on
%x0=1, size, patient age, and bloodflow rate
clear
global x y;
x=[1,7,50,42;1,77,77,2;1,87,75,1;1,87,70,21;1,60,91,10;1,80,90,3;1,30,20,23;1,10,25,20;1,15,20,33;1,10,20,23;1,50,60,1;1,10,20,15;1,40,70,5;1,10,10,24;1,80,80,1];
y=[0;1;1;1;1;1;0;0;0;0;1;0;1;0;1];
fminsearch(@cost,[-1,.1,.1,.1])


function f=cost(a)
global x y;
f=0;
e=2.71828;
for i=1:length(y)
    f=f+(y(i)*log(1./(1+e^(-a*x(i,:)'))));
    f=f+((1-y(i))*log(1-(1./(1+e^(-a*x(i,:)')))));
end
f=-f./length(y);
end
```


Note that you have 4 features and only 6 data points. It is very likely the perfect separation was happening in your example.
To fix this, you can add more data points (use the whole data instead of just some samples), reduce number of features, or add regularization.
One of my answers have the demo of the problem and why regularization will help.
Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?
