Answers.Body: <p>Just realised the issue - super subtle (at least for a Python novice like me) - I implemented numerical gradient checking (as I should have done from the start) and saw the gradient descent was working incorrectly.</p>
  <p>The dataset Y was created as a datatype &quot;uint8&quot;. In the line of the back-propagation where I calculate the derivative with respect to &quot;a&quot; (&quot;da&quot;) the statement &quot;-Y&quot; was returning 255 for those values of Y that were 1 - this appears to be a feature of these unsigned binary numbers.</p>
  <p>By replacing &quot;uint8&quot; with &quot;int8&quot; in the definition of Y - or else forcing the correct behaviour by writing &quot;-1.*Y&quot; in the calculation of &quot;da&quot; (which I think casts &quot;Y&quot; as an integer rather than an unsigned integer) I managed to produce the correct behaviour.</p>
  <p>This in turn means the gradient descent is doing what it should and the logistic regression converges to a stable value of the cost. Accuracy then tracks cost as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T15:11:33.587Z
Answers.LastActivityDate: 2020-09-16T15:11:33.587Z
Answers.UserDisplayName: a_former_scientist
Body: <p><strong>Background</strong> I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see picture below). Of course, it totally fails; in Andrew Ng's course, the failure of logistic regression to fit to this motivates developing a neural net - which works quite nicely.  But my question concerns what my logistic model is doing and why.</p>
  <p><a href="https://i.sstatic.net/EUG4P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EUG4P.png" alt="non-linear data set" /></a></p>
  <p><strong>The problem</strong> My logistic regression model's cost increases, even after massively reducing the learning rate. But at the same time my accuracy (slowly) increases. I simply cannot see why.</p>
  <p>To confuse matters even more - if I resort to a negative learning rate (essentially trying to force the calibration to higher cost values) the cost then decreases for a time until the accuracy hits 50%. After this point, the cost then inexorably increases - but the accuracy stays equal to 50%. The solution so found is to set all points to either red or blue (a reasonable fit given logistic regression simply cannot work on this data).</p>
  <p><strong>My questions and thoughts on answers</strong> I have reproduced the Python code below - hopefully it's clear. My questions are:</p>
  <ol>
  <li>Is there a mistake in the model that explains why negative learning rates seem to work better?</li>
  <li>On the topic of why the cost increases even as accuracy asymptotes to 50%: is the issue that once the model has discovered the &quot;all points equal to either red or blue&quot; solution the parameters &quot;w&quot; and &quot;b&quot; just get larger and larger (in absolute terms) - driving all of the predictions closer to 1 (or conversely if it predicts all points are 0)?</li>
  </ol>
  <p>To explain this second question a bit more: imagine red points are defined by y = 1. Suppose parameters w, b are chosen such that the probability for every point equals 0.9. Then the model predicts all points are red - which is correct for half the points. The model can then improve half the predictions by driving w and b up (so that sigmoid ( w*x + b) --&gt; 1). But of course, this makes half the predictions (the blue points) more and more wrong - which causes the cost function for those points - log(1 - prob) - to diverge. I don't truly see why gradient descent would do this but it's all I can think of for the peculiar behaviour of the algorithm.</p>
  <p>Hope this all makes sense. Hit me up if not.</p>
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  
  
  # function to create a flower-like arrangement of 1s and 0s
  def load_planar_dataset():
      np.random.seed(1)
      m = 400 # number of examples
      N = int(m/2) # number of points per class
      D = 2 # dimensionality / i.e. work in 2d plane - so X is a set of (x,y) coordinate points
      X = np.zeros((m,D)) # data matrix where each row is a single example
      Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
      a = 4 # maximum ray of the flower
  
      for j in range(2):
          ix = range(N*j,N*(j+1))
          t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta / random element mixes up some of the petals so you get mostly blue with some red petals and vice-versa
          r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius / again random element alters  shape of flower slightly
          X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
          Y[ix] = j
          
      X = X.T # transpose so columns = training example as per standard in lectures
      Y = Y.T
  
      return X, Y
  
  # function to plot the above data plus a modelled decision boundary - works by applying model to grid of points and colouring accordingly
  def plot_decision_boundary(model, X, y):
      # Set min and max values and give it some padding
      x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
      y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
      h = 0.01
      # Generate a grid of points with distance h between them
      xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
      # Predict the function value for the whole grid
      Z = model(np.c_[xx.ravel(), yy.ravel()])
      Z = Z.reshape(xx.shape)
      # Plot the contour and training examples
      plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
      plt.ylabel('x2')
      plt.xlabel('x1')
      plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
  
  
  # sigmoid function as per sandard linear regression
  def sigmoid(z):
      &quot;&quot;&quot;
      Compute the sigmoid of z
  
      Arguments:
      z -- A scalar or numpy array of any size.
  
      Return:
      s -- sigmoid(z)
      &quot;&quot;&quot;
  
      s = 1. / (1. + np.exp(-z))
      
      return s
  
  
  # 
  def propagate(w, b, X, Y):
      &quot;&quot;&quot;
      Implement the cost function and its gradient for the propagation explained above
  
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of size (num_px * num_px * 3, number of examples)
      Y -- true &quot;label&quot; vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)
  
      Return:
      cost -- negative log-likelihood cost for logistic regression
      dw -- gradient of the loss with respect to w, thus same shape as w
      db -- gradient of the loss with respect to b, thus same shape as b
      &quot;&quot;&quot;
  
      m = X.shape[1];
  
      # forward prop
      Z = np.dot(w.T, X) + b; 
      A = sigmoid(Z); # activiation = the prediction of the model
  
  
      # compute cost
      cost =  - 1. / m * np.sum( (Y * np.log(A) + (1. - Y) * np.log(1. - A)  ) )
  
      #back prop for gradient descent
  
      da = - Y / A + (1. - Y) / (1. - A)  
      dz = da * A * (1. - A) # = - Y (1-A) + (1. - Y) A =  A - Y  
      dw = 1. / m * np.dot( X, dz.T )
      db = 1. / m * np.sum(dz)
  
      grads = {&quot;dw&quot;: dw,
                  &quot;db&quot;: db}
  
      return grads, cost
  
  
  def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
      &quot;&quot;&quot;
      This function optimizes w and b by running a gradient descent algorithm
      
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of shape (num_px * num_px * 3, number of examples)
      Y -- true &quot;label&quot; vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
      num_iterations -- number of iterations of the optimization loop
      learning_rate -- learning rate of the gradient descent update rule
      print_cost -- True to print the loss every 100 steps
      
      Returns:
      params -- dictionary containing the weights w and bias b
      grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
      costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
      
      &quot;&quot;&quot;
      costs = []
  
      for i in range(num_iterations):
          
          # cost /gradient calculation
          grads, cost = propagate(w, b, X, Y)
  
          #retrieve derivatives
          dw = grads[&quot;dw&quot;]
          db = grads[&quot;db&quot;]
  
          # update values according to gradient descent algorithm
          w = w - learning_rate * dw
          b = b - learning_rate * db
  
          # record the costs
          if i % 100 == 0:
              costs.append(cost)
  
              # Print the cost every 100 training iterations
              if print_cost:
                  print(&quot;Cost after iteration %i: %f&quot; %(i, cost))
  
  
  
      params = {  &quot;w&quot;: w,
                  &quot;b&quot;: b}
  
      grads = {   &quot;dw&quot;: dw,
                  &quot;db&quot;: db}
  
      return params, grads, costs
  
  
  def predict(w, b, X):
      '''
      Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
      
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of size (num_px * num_px * 3, number of examples)
      
      Returns:
      Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
      '''
      Z = np.dot(w.T, X) + b
      A = sigmoid(Z)
  
  
      Y_prediction = (A &gt;= 0.5).astype(int)
  
      return Y_prediction
  
  
  
  
  np.random.seed(1) # set a seed so that the results are consistent
  
  X, Y = load_planar_dataset()
  
  # Visualize the data:
  
  plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral); # s = size of points; cmap are nicer colours
  plt.show()
  
  shape_X = X.shape
  shape_Y = Y.shape
  m = shape_Y[1]  # training set size
  n = shape_X[0] # number of features (2)
  
  
  # initialise parameters
  w = np.random.rand(n, 1)
  b = 0
  
  # print accuracy of initial parameters by comparing prediction to 
  print(&quot;train accuracy: {} %&quot;.format(100 - np.mean(np.abs(predict(w, b, X) - Y)) * 100))
  
  
  # fit model and print out costs every 100 iterations of the forward / back prop
  parameters, grads, costs = optimize(w, b, X, Y, num_iterations = 10000, learning_rate = 0.000005, print_cost = True)
  
  
  # return the prediction
  Y_prediction = predict(parameters[&quot;w&quot;], parameters[&quot;b&quot;], X)
  
  # print accuracy of fitted model
  print(&quot;train accuracy: {} %&quot;.format(100 - np.mean(np.abs(Y_prediction - Y)) * 100))
  
  
  # print parameters for interest
  print( parameters[&quot;w&quot;] , parameters[&quot;b&quot;] )
  
  # plot decision boundary
  plot_decision_boundary(lambda x: predict(parameters[&quot;w&quot;], parameters[&quot;b&quot;], x.T), X, Y)
  plt.show()
  
  List item
  </code></pre>
  
Tags: <deep-learning><backpropagation><objective-functions><logistic-regression>
Title: Back propagation approach to logistic regression: why is cost diverging but accuracy increasing?
UserDisplayName: a_former_scientist
ThreadText: # Back propagation approach to logistic regression: why is cost diverging but accuracy increasing?
  
  (Asked by: username_0 on 2020-09-16)
  
  Background I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see picture below). Of course, it totally fails; in Andrew Ng's course, the failure of logistic regression to fit to this motivates developing a neural net - which works quite nicely.  But my question concerns what my logistic model is doing and why.
  
  The problem My logistic regression model's cost increases, even after massively reducing the learning rate. But at the same time my accuracy (slowly) increases. I simply cannot see why.
  To confuse matters even more - if I resort to a negative learning rate (essentially trying to force the calibration to higher cost values) the cost then decreases for a time until the accuracy hits 50%. After this point, the cost then inexorably increases - but the accuracy stays equal to 50%. The solution so found is to set all points to either red or blue (a reasonable fit given logistic regression simply cannot work on this data).
  My questions and thoughts on answers I have reproduced the Python code below - hopefully it's clear. My questions are:
  
  Is there a mistake in the model that explains why negative learning rates seem to work better?
  On the topic of why the cost increases even as accuracy asymptotes to 50%: is the issue that once the model has discovered the "all points equal to either red or blue" solution the parameters "w" and "b" just get larger and larger (in absolute terms) - driving all of the predictions closer to 1 (or conversely if it predicts all points are 0)?
  
  To explain this second question a bit more: imagine red points are defined by y = 1. Suppose parameters w, b are chosen such that the probability for every point equals 0.9. Then the model predicts all points are red - which is correct for half the points. The model can then improve half the predictions by driving w and b up (so that sigmoid ( w*x + b) --> 1). But of course, this makes half the predictions (the blue points) more and more wrong - which causes the cost function for those points - log(1 - prob) - to diverge. I don't truly see why gradient descent would do this but it's all I can think of for the peculiar behaviour of the algorithm.
  Hope this all makes sense. Hit me up if not.
  import numpy as np
  import matplotlib.pyplot as plt
  
  
  # function to create a flower-like arrangement of 1s and 0s
  def load_planar_dataset():
      np.random.seed(1)
      m = 400 # number of examples
      N = int(m/2) # number of points per class
      D = 2 # dimensionality / i.e. work in 2d plane - so X is a set of (x,y) coordinate points
      X = np.zeros((m,D)) # data matrix where each row is a single example
      Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
      a = 4 # maximum ray of the flower
  
      for j in range(2):
          ix = range(N*j,N*(j+1))
          t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta / random element mixes up some of the petals so you get mostly blue with some red petals and vice-versa
          r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius / again random element alters  shape of flower slightly
          X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
          Y[ix] = j
          
      X = X.T # transpose so columns = training example as per standard in lectures
      Y = Y.T
  
      return X, Y
  
  # function to plot the above data plus a modelled decision boundary - works by applying model to grid of points and colouring accordingly
  def plot_decision_boundary(model, X, y):
      # Set min and max values and give it some padding
      x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
      y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
      h = 0.01
      # Generate a grid of points with distance h between them
      xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
      # Predict the function value for the whole grid
      Z = model(np.c_[xx.ravel(), yy.ravel()])
      Z = Z.reshape(xx.shape)
      # Plot the contour and training examples
      plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
      plt.ylabel('x2')
      plt.xlabel('x1')
      plt.scatter(X[0, :], X[1, :], c=y, cmap=plt.cm.Spectral)
  
  
  # sigmoid function as per sandard linear regression
  def sigmoid(z):
      """
      Compute the sigmoid of z
  
      Arguments:
      z -- A scalar or numpy array of any size.
  
      Return:
      s -- sigmoid(z)
      """
  
      s = 1. / (1. + np.exp(-z))
      
      return s
  
  
  # 
  def propagate(w, b, X, Y):
      """
      Implement the cost function and its gradient for the propagation explained above
  
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of size (num_px * num_px * 3, number of examples)
      Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)
  
      Return:
      cost -- negative log-likelihood cost for logistic regression
      dw -- gradient of the loss with respect to w, thus same shape as w
      db -- gradient of the loss with respect to b, thus same shape as b
      """
  
      m = X.shape[1];
  
      # forward prop
      Z = np.dot(w.T, X) + b; 
      A = sigmoid(Z); # activiation = the prediction of the model
  
  
      # compute cost
      cost =  - 1. / m * np.sum( (Y * np.log(A) + (1. - Y) * np.log(1. - A)  ) )
  
      #back prop for gradient descent
  
      da = - Y / A + (1. - Y) / (1. - A)  
      dz = da * A * (1. - A) # = - Y (1-A) + (1. - Y) A =  A - Y  
      dw = 1. / m * np.dot( X, dz.T )
      db = 1. / m * np.sum(dz)
  
      grads = {"dw": dw,
                  "db": db}
  
      return grads, cost
  
  
  def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
      """
      This function optimizes w and b by running a gradient descent algorithm
      
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of shape (num_px * num_px * 3, number of examples)
      Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
      num_iterations -- number of iterations of the optimization loop
      learning_rate -- learning rate of the gradient descent update rule
      print_cost -- True to print the loss every 100 steps
      
      Returns:
      params -- dictionary containing the weights w and bias b
      grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
      costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
      
      """
      costs = []
  
      for i in range(num_iterations):
          
          # cost /gradient calculation
          grads, cost = propagate(w, b, X, Y)
  
          #retrieve derivatives
          dw = grads["dw"]
          db = grads["db"]
  
          # update values according to gradient descent algorithm
          w = w - learning_rate * dw
          b = b - learning_rate * db
  
          # record the costs
          if i % 100 == 0:
              costs.append(cost)
  
              # Print the cost every 100 training iterations
              if print_cost:
                  print("Cost after iteration %i: %f" %(i, cost))
  
  
  
      params = {  "w": w,
                  "b": b}
  
      grads = {   "dw": dw,
                  "db": db}
  
      return params, grads, costs
  
  
  def predict(w, b, X):
      '''
      Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
      
      Arguments:
      w -- weights, a numpy array of size (num_px * num_px * 3, 1)
      b -- bias, a scalar
      X -- data of size (num_px * num_px * 3, number of examples)
      
      Returns:
      Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
      '''
      Z = np.dot(w.T, X) + b
      A = sigmoid(Z)
  
  
      Y_prediction = (A >= 0.5).astype(int)
  
      return Y_prediction
  
  
  
  
  np.random.seed(1) # set a seed so that the results are consistent
  
  X, Y = load_planar_dataset()
  
  # Visualize the data:
  
  plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral); # s = size of points; cmap are nicer colours
  plt.show()
  
  shape_X = X.shape
  shape_Y = Y.shape
  m = shape_Y[1]  # training set size
  n = shape_X[0] # number of features (2)
  
  
  # initialise parameters
  w = np.random.rand(n, 1)
  b = 0
  
  # print accuracy of initial parameters by comparing prediction to 
  print("train accuracy: {} %".format(100 - np.mean(np.abs(predict(w, b, X) - Y)) * 100))
  
  
  # fit model and print out costs every 100 iterations of the forward / back prop
  parameters, grads, costs = optimize(w, b, X, Y, num_iterations = 10000, learning_rate = 0.000005, print_cost = True)
  
  
  # return the prediction
  Y_prediction = predict(parameters["w"], parameters["b"], X)
  
  # print accuracy of fitted model
  print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction - Y)) * 100))
  
  
  # print parameters for interest
  print( parameters["w"] , parameters["b"] )
  
  # plot decision boundary
  plot_decision_boundary(lambda x: predict(parameters["w"], parameters["b"], x.T), X, Y)
  plt.show()
  
  List item
  
  **Tags:** deep-learning, backpropagation, objective-functions, logistic-regression
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-09-16. Score: 1
  
  Just realised the issue - super subtle (at least for a Python novice like me) - I implemented numerical gradient checking (as I should have done from the start) and saw the gradient descent was working incorrectly.
  
  The dataset Y was created as a datatype "uint8". In the line of the back-propagation where I calculate the derivative with respect to "a" ("da") the statement "-Y" was returning 255 for those values of Y that were 1 - this appears to be a feature of these unsigned binary numbers.
  
  By replacing "uint8" with "int8" in the definition of Y - or else forcing the correct behaviour by writing "-1.\*Y" in the calculation of "da" (which I think casts "Y" as an integer rather than an unsigned integer) I managed to produce the correct behaviour.
  
  This in turn means the gradient descent is doing what it should and the logistic regression converges to a stable value of the cost. Accuracy then tracks cost as well.
  
  
  ---
  
  