Answers.Body: <p>So, firstly, for <span class="math-container">$h_{\Theta}(x)$</span> to be <span class="math-container">$1$</span>, the weighted sum of <span class="math-container">$x$</span> (after you dot product it with <span class="math-container">$\Theta$</span>) would have to be literally infinity, if you're using the sigmoid function. Doesn't happen in practice, even with the rounding computers do, as we don't use big numbers to initialize our <span class="math-container">$\Theta$</span> matrices.</p>
  <p>Intuitively, that'd mean you're basically more certain than one can possibly be in this universe that the label of this example should be <span class="math-container">$1$</span>.</p>
  <p>So, if <span class="math-container">$(1 - h_{\Theta}(x)) = 0$</span>, <span class="math-container">$y$</span> is <em>certainly</em> <span class="math-container">$1$</span>, and so <span class="math-container">$1-y$</span> <em>will</em> be zero.</p>
  <p>Secondly, the convention is to drop the entire right-hand-side term when <span class="math-container">$y^{(i)}$</span> is <span class="math-container">$1$</span>. This will not cause problems when programming, due to the first point I made above.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-23T06:55:12.027Z
Answers.LastActivityDate: 2021-12-12T08:42:09.180Z
Answers.LastEditDate: 2021-12-12T08:42:09.180Z
Answers.UserDisplayName: Avik Mohan
Body: <p>I am implementing a neural network to train it on handwritten digits.</p>
  <p>Here is the cost function that I am implementing.</p>
  <p><span class="math-container">$$J(\Theta)=-\frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K}\left[y_{k}^{(i)} \log \left(\left(h_{\Theta}\left(x^{(i)}\right)\right)_{k}\right)+\left(1-y_{k}^{(i)}\right) \log \left(1-\left(h_{\Theta}\left(x^{(i)}\right)\right)_{k}\right)\right]+ \\\frac{\lambda}{2 m} \sum_{l=1}^{L-1} \sum_{i=1}^{s_{l}} \sum_{j=1}^{s_{l+1}}\left(\Theta_{j, i}^{(l)}\right)^{2}$$</span></p>
  <p>In <span class="math-container">$\log(1-(h(x))$</span>, if <span class="math-container">$h(x)$</span> is <span class="math-container">$1$</span>, then it would result in <span class="math-container">$\log(1-1)= \log(0)$</span>. So, I'm getting a math error.</p>
  <p>I'm initializing the weights randomly between 10-60. I'm not sure where I have to change and what I have to change.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-01-23T03:53:47.087Z
Comments.Text: what if 1-y is not zero and 1-(h(x) is zero?
Comments.UserDisplayName: Gokulakannan
Tags: <neural-networks><machine-learning><objective-functions><implementation><sigmoid>
Title: How do I avoid the "math domain error" when the input to the log is zero in the objective function of a neural network?
UserDisplayName: Gokulakannan
ThreadText: # How do I avoid the "math domain error" when the input to the log is zero in the objective function of a neural network?
  
  (Asked by: username_0 on 2018-01-22)
  
  I am implementing a neural network to train it on handwritten digits.
  
  Here is the cost function that I am implementing.
  
  $$J(\Theta)=-\frac{1}{m} \sum\_{i=1}^{m} \sum\_{k=1}^{K}\left[y\_{k}^{(i)} \log \left(\left(h\_{\Theta}\left(x^{(i)}\right)\right)\_{k}\right)+\left(1-y\_{k}^{(i)}\right) \log \left(1-\left(h\_{\Theta}\left(x^{(i)}\right)\right)\_{k}\right)\right]+ \\\frac{\lambda}{2 m} \sum\_{l=1}^{L-1} \sum\_{i=1}^{s\_{l}} \sum\_{j=1}^{s\_{l+1}}\left(\Theta\_{j, i}^{(l)}\right)^{2}$$
  
  In $\log(1-(h(x))$, if $h(x)$ is $1$, then it would result in $\log(1-1)= \log(0)$. So, I'm getting a math error.
  
  I'm initializing the weights randomly between 10-60. I'm not sure where I have to change and what I have to change.
  
  **Tags:** neural-networks, machine-learning, objective-functions, implementation, sigmoid
  
  > **Comments:**
  > **username_0**: what if 1-y is not zero and 1-(h(x) is zero?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-23. Score: 1 | Accepted answer
  
  So, firstly, for $h\_{\Theta}(x)$ to be $1$, the weighted sum of $x$ (after you dot product it with $\Theta$) would have to be literally infinity, if you're using the sigmoid function. Doesn't happen in practice, even with the rounding computers do, as we don't use big numbers to initialize our $\Theta$ matrices.
  
  Intuitively, that'd mean you're basically more certain than one can possibly be in this universe that the label of this example should be $1$.
  
  So, if $(1 - h\_{\Theta}(x)) = 0$, $y$ is *certainly* $1$, and so $1-y$ *will* be zero.
  
  Secondly, the convention is to drop the entire right-hand-side term when $y^{(i)}$ is $1$. This will not cause problems when programming, due to the first point I made above.
  
  
  ---
  
  