Answers.Body: <p>We can analyze a basic common example: approximation of AND logic gate by a NN. </p>
  
  <p>The inputs to the NN will be "x1" and "x2" and its output is "y". The data to be learned by the NN is:</p>
  
  <p><a href="https://i.sstatic.net/Uxvjb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Uxvjb.png" alt="enter image description here"></a></p>
  
  <p>The basic NN has one intermediate cell with activation function sigmoid and one output cell identity function. That means:</p>
  
  <p><a href="https://i.sstatic.net/1twPv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1twPv.png" alt="enter image description here"></a></p>
  
  <p>(note by symmetry we assume w1=w2)</p>
  
  <p>Then, the error is:</p>
  
  <p><a href="https://i.sstatic.net/rSSsf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rSSsf.png" alt="enter image description here"></a> </p>
  
  <p>These are some plots of this function:</p>
  
  <p><a href="https://i.sstatic.net/HOfuo.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HOfuo.jpg" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/AltDb.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AltDb.jpg" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/nAm7q.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nAm7q.jpg" alt="enter image description here"></a></p>
  
  <p>And this is the w derivate of e :</p>
  
  <p><a href="https://i.sstatic.net/wVCIj.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wVCIj.jpg" alt="enter image description here"></a></p>
  
  <p>Sage Math code (<a href="http://sagecell.sagemath.org/" rel="nofollow noreferrer">link</a>) for this graph:</p>
  
  <pre><code>s(x)=1/(1+e^(-1*x))
  e(w,b)=(1/2)*(s(b)^2+2*s(w+b)^2+(1-s(2*w+b))^2)
  plot3d(e(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))
  
  edw=e.derivative(w)
  plot3d(edw(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))
  </code></pre>
  
  <p>(PS: please, activate latex on this stack exchange).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-16T10:16:48.550Z
Answers.LastActivityDate: 2018-03-16T10:39:42.203Z
Answers.LastEditDate: 2018-03-16T10:39:42.203Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for weight change, for the purpose of reducing the Loss function in a Linear Regression Model, determined by this algorithm?</p>
  
Tags: <machine-learning><linear-regression><gradient-descent>
Title: How is direction of weight change determined by Gradient Descent algorithm
UserDisplayName: Amey
ThreadText: # How is direction of weight change determined by Gradient Descent algorithm
  
  (Asked by: username_0 on 2018-03-14)
  
  The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for weight change, for the purpose of reducing the Loss function in a Linear Regression Model, determined by this algorithm?
  
  **Tags:** machine-learning, linear-regression, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-03-16. Score: 0
  
  We can analyze a basic common example: approximation of AND logic gate by a NN.
  
  The inputs to the NN will be "x1" and "x2" and its output is "y". The data to be learned by the NN is:
  
  [![enter image description here](https://i.sstatic.net/Uxvjb.png)](https://i.sstatic.net/Uxvjb.png)
  
  The basic NN has one intermediate cell with activation function sigmoid and one output cell identity function. That means:
  
  [![enter image description here](https://i.sstatic.net/1twPv.png)](https://i.sstatic.net/1twPv.png)
  
  (note by symmetry we assume w1=w2)
  
  Then, the error is:
  
  [![enter image description here](https://i.sstatic.net/rSSsf.png)](https://i.sstatic.net/rSSsf.png)
  
  These are some plots of this function:
  
  [![enter image description here](https://i.sstatic.net/HOfuo.jpg)](https://i.sstatic.net/HOfuo.jpg)
  [![enter image description here](https://i.sstatic.net/AltDb.jpg)](https://i.sstatic.net/AltDb.jpg)
  [![enter image description here](https://i.sstatic.net/nAm7q.jpg)](https://i.sstatic.net/nAm7q.jpg)
  
  And this is the w derivate of e :
  
  [![enter image description here](https://i.sstatic.net/wVCIj.jpg)](https://i.sstatic.net/wVCIj.jpg)
  
  Sage Math code ([link](http://sagecell.sagemath.org/)) for this graph:
  
  ```
  s(x)=1/(1+e^(-1*x))
  e(w,b)=(1/2)*(s(b)^2+2*s(w+b)^2+(1-s(2*w+b))^2)
  plot3d(e(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))
  
  edw=e.derivative(w)
  plot3d(edw(w,b),(w,-5,5),(b,-5,5),adaptive=True, color=rainbow(60, 'rgbtuple'))
  
  ```
  
  (PS: please, activate latex on this stack exchange).
  
  
  ---
  
  