Body: <p>So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ReLUs) and the last output layer will be linear because we are trying to catch a specific number and not a probability. I've done the forward propagation calculations but I'm stuck at the back propagation ones.</p>
  <p>Let's say that I'm gonna use the cross entropy loss function: (we will implement the MSE as well)   <span class="math-container">$-(y \log (a)+(1-y) \log (1-a))$</span>. (I understand that this is not a good option for a regression problem)</p>
  <p>So we can easily find the dJ/dA <span class="math-container">$d A=\frac{\partial J}{\partial A}=-\left(\frac{y}{A}-\frac{1-y}{1-A}\right)$</span> of the last layer and we can start going backwards finding the <span class="math-container">$\frac{\partial J}{\partial Z^{[L]}}$</span> which we can calculate from the equation: <span class="math-container">$d Z^{[L]}=d A*g^{\prime}\left(Z^{[L]}\right)$</span> The problem lies at the second part of this equation where the derivative of g is.</p>
  <p>What will be the outcome since we have a linear activation function which derivative is equal with 1? (Activation function: f(x) = x, f'(x) = 1)</p>
  <p>Will it be an identity matrix with the shape of Z[L] or a matrix full of ones with the same shape again? I'm asking about the term <span class="math-container">$g^{\prime}\left(Z^{[L]}\right)$</span>.</p>
  <p>Many thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-13T10:00:03.243Z
Comments.Text: You don't want to implement cross entropy loss with linear activation. The cross entropy loss function is poorly defined when inputs are less than 0 or greated than 1. It can be done though, so if you are still interested in the theory, can you please specify in more detail what the loss function is - give a formula, because there are a couple of variations that are called "cross entropy".
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-13T10:17:10.247Z
Comments.Text: You can use LaTex between `$` signs e.g. `$\nabla_{Z^{[L]}} = \frac{\partial J}{\partial Z^{[L]}}$` becomes $\nabla_{Z^{[L]}} J = \frac{\partial J}{\partial Z^{[L]}}$. I suggest you do this in the question to make it easier to read. Use [edit] link to alter the question. Also, can you be more specific about what "the second part of this equation" refers to - say which term you mean? Also be clear about "outcome" - are you asking about the overall loss value, or just the value of the term yo uare having trouble with?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-13T15:07:13.197Z
Comments.Text: I just edited the post. You can have a second read now. Many thanks.
Comments.UserDisplayName: ChrisP
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-13T20:38:17.463Z
Comments.Text: Can you please put your main question in the title? Right now, your title is not a question and it's not very descriptive.
Comments.UserDisplayName: nbro
Tags: <neural-networks><backpropagation><linear-regression>
Title: Linear output layer back propagation
UserDisplayName: ChrisP
ThreadText: # Linear output layer back propagation
  
  (Asked by: username_0 on 2020-10-13)
  
  So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ReLUs) and the last output layer will be linear because we are trying to catch a specific number and not a probability. I've done the forward propagation calculations but I'm stuck at the back propagation ones.
  
  Let's say that I'm gonna use the cross entropy loss function: (we will implement the MSE as well) $-(y \log (a)+(1-y) \log (1-a))$. (I understand that this is not a good option for a regression problem)
  
  So we can easily find the dJ/dA $d A=\frac{\partial J}{\partial A}=-\left(\frac{y}{A}-\frac{1-y}{1-A}\right)$ of the last layer and we can start going backwards finding the $\frac{\partial J}{\partial Z^{[L]}}$ which we can calculate from the equation: $d Z^{[L]}=d A\*g^{\prime}\left(Z^{[L]}\right)$ The problem lies at the second part of this equation where the derivative of g is.
  
  What will be the outcome since we have a linear activation function which derivative is equal with 1? (Activation function: f(x) = x, f'(x) = 1)
  
  Will it be an identity matrix with the shape of Z[L] or a matrix full of ones with the same shape again? I'm asking about the term $g^{\prime}\left(Z^{[L]}\right)$.
  
  Many thanks.
  
  **Tags:** neural-networks, backpropagation, linear-regression
  > **Comments:**
  > **username_1**: You don't want to implement cross entropy loss with linear activation. The cross entropy loss function is poorly defined when inputs are less than 0 or greated than 1. It can be done though, so if you are still interested in the theory, can you please specify in more detail what the loss function is - give a formula, because there are a couple of variations that are called "cross entropy".  
  > **username_1**: You can use LaTex between `$` signs e.g. `$\nabla_{Z^{[L]}} = \frac{\partial J}{\partial Z^{[L]}}$` becomes $\nabla_{Z^{[L]}} J = \frac{\partial J}{\partial Z^{[L]}}$. I suggest you do this in the question to make it easier to read. Use [edit] link to alter the question. Also, can you be more specific about what "the second part of this equation" refers to - say which term you mean? Also be clear about "outcome" - are you asking about the overall loss value, or just the value of the term yo uare having trouble with?  
  > **username_0**: I just edited the post. You can have a second read now. Many thanks.  
  > **username_2**: Can you please put your main question in the title? Right now, your title is not a question and it's not very descriptive.  
  
  ---
  
  ## 0 answers
  
  