Answers.Body: <h1>Lets mock some data up.</h1>
  
  <blockquote>
    <p>"100 numbers, each one is a parameter, they together define a number X(also given)"</p>
  </blockquote>
  
  <pre><code># i.e. size of X_train -&gt; [n x d]
  # i.e. size of X_train -&gt; [??? x 100]  , when d = 100
  
  # "I have 20000 instances for training"
  # i.e. size of X_train -&gt; [20000 x 100], when n = 20000
  
  import torch
  import numpy as np
  
  X_train = torch.rand((20000, 100))
  X_train = np.random.rand(20000, 100) # Or using numpy
  </code></pre>
  
  <h1>But what is your Y?</h1>
  
  <pre><code># Since the definition of a regression task, 
  # loosely means to predict an output real number
  # given an input of d dimension
  
  # So the appropriate Y_train would 
  # be of dimension [n x 1] 
  # and look like this:
  
  y_train = torch.rand((20000, 1))
  
  y_train = np.random.rand(20000, 1) # Or using numpy
  </code></pre>
  
  <h1>What is a linear perceptron?</h1>
  
  <p>Taking definition from <a href="https://github.com/alvations/tsundoku/blob/master/completed/Session%201%20-%20Hello%20DL%20World.ipynb" rel="nofollow noreferrer">this tutorial</a></p>
  
  <p><a href="https://i.sstatic.net/eEVd5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eEVd5.png" alt="enter image description here"></a></p>
  
  <h1>Thus, in picture:</h1>
  
  <p><a href="https://i.sstatic.net/ZPM3A.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZPM3A.png" alt="enter image description here"></a></p>
  
  <h1>Next we need to define training routine,</h1>
  
  <p>For now take it as biblical truth that this is an okay routine to train a neural net model (this isn't the only way but easiest or supervised learning):</p>
  
  <p><a href="https://i.sstatic.net/4Jt7M.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4Jt7M.png" alt="enter image description here"></a></p>
  
  <h1>In code:</h1>
  
  <pre><code>import math
  import numpy as np
  np.random.seed(0)
  
  def sigmoid(x): # Returns values that sums to one.
      return 1 / (1 + np.exp(-x))
  
  def sigmoid_derivative(sx): 
      # See https://math.stackexchange.com/a/1225116
      # Hint: let sx = sigmoid(x)
      return sx * (1 - sx)
  
  def cost(predicted, truth):
      return np.abs(truth - predicted)
  
  
  num_epochs = 10000 # No. of times to iterate.
  learning_rate = 0.03 # How large a step to take per iteration.
  
  # Lets standardize and call our inputs X and outputs Y
  X = np.array(torch.rand((20000, 100)))
  Y = or_output
  
  for _ in range(num_epochs):
      layer0 = X
  
      # Step 2a: Multiply the weights vector with the inputs, sum the products, i.e. s
      # Step 2b: Put the sum through the sigmoid, i.e. f()
      # Inside the perceptron, Step 2. 
      layer1 = sigmoid(np.dot(X, W))
  
      # Back propagation.
      # Step 3a: Compute the errors, i.e. difference between expected output and predictions
      # How much did we miss?
      layer1_error = cost(layer1, Y)
  
      # Step 3b: Multiply the error with the derivatives to get the delta
      # multiply how much we missed by the slope of the sigmoid at the values in layer1
      layer1_delta = layer1_error * sigmoid_derivative(layer1)
  
      # Step 3c: Multiply the delta vector with the inputs, sum the product (use np.dot)
      # Step 4: Multiply the learning rate with the output of Step 3c.
      W +=  learning_rate * np.dot(layer0.T, layer1_delta)
  </code></pre>
  
  <p>Now that we learn the model, i.e. the <code>W</code>. </p>
  
  <p>When we see the data points that we need to use the model on, we apply the same forward propagation step, i.e. <code>layer1 = sigmoid(np.dot(X, W))</code> </p>
  
  <p>Since we have:</p>
  
  <blockquote>
    <p>I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.</p>
  </blockquote>
  
  <p>And in code:</p>
  
  <pre><code># If we mock up the data,
  # it should be the same internal dimension. 
  X_test = np.random.rand(5000, 100)
  
  # The desired output just needs to pass through the W and the activation:
  # the shape of `output` -&gt; [5000 x 1] , 
  # where there's 1 output value for each input.
  output = sigmoid(np.dot(X_test, W))
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T16:00:00.600Z
Answers.Text: There's some weird things happening here when we design our network with a sigmoid, we naturally force the output values to be between the `(0, 1)` range. But this is partly because when we mock up the data using `np.random.rand`, the values are expected to fall between the `(0,1)`.
Answers.UserDisplayName: alvas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T16:01:55.620Z
Answers.Text: A normal regression task where the output has arbitrary range of values, simply skip the sigmoid layer for the forward propagation and then the error derivatives in the `sigmoid_derivative(layer1)` would just be 1.
Answers.UserDisplayName: alvas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T15:58:26.083Z
Answers.LastActivityDate: 2020-05-11T15:58:26.083Z
Answers.UserDisplayName: alvas
Answers.Body: <p>The quick answer is that you want to use an activation function on the output layer that does not compress values to <span class="math-container">$(0,1)$</span>. Depending on your software, this might be called "linear" or "identity". It looks like Keras just wants you to leave off the activation function: <code>model.add(Dense(1))</code>.</p>
  
  <p>The typical way of thinking of a neural network as a classifier (let's say a binary classifier) is just extending a logistic regression. In fact, when you use a sigmoid activation function on the output node, you're (sort of) running logistic regression on the final hidden layer. </p>
  
  <p>A logistic regression is one type of generalized linear model. The gist of GLM is that some transformation of the the value of interest is a linear function of the feature space.</p>
  
  <p>Let <span class="math-container">$X$</span> be the data matrix for the feature space. Let <span class="math-container">$\beta$</span> be a parameter vector. Then <span class="math-container">$\hat{y} = \mathbb{E}[y] = X\beta$</span> is the linear model, and <span class="math-container">$g(\mathbb{E}[y]) = X\beta$</span> is the generalized linear model (vectorized, so apply <span class="math-container">$g$</span> to each <span class="math-container">$y_i$</span>).</p>
  
  <p>But we could extend this to a nonlinear transformation, and when a neural network is a binary classifier, this is precisely what we're doing. Instead of the transformation of <span class="math-container">$X$</span> being given by <span class="math-container">$\beta$</span> and thus linear, we apply some nonlinear transformation <span class="math-container">$f$</span> and get <span class="math-container">$g(\mathbb{E}[y]) = f(X)$</span>.</p>
  
  <p>The terminology in GLM is "link function", but that is essentially the activation function on the final node(s) of the neural network. Consequently, all of the GLM link functions are in play, and one of those link functions is the identity function. For a GLM, that's just linear regression. For your neural network, it will be a neural network (nonlinear) regression, which sounds like what you want.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T18:32:46.193Z
Answers.LastActivityDate: 2020-05-11T18:41:10.433Z
Answers.LastEditDate: 2020-05-11T18:41:10.433Z
Answers.UserDisplayName: Dave
Answers.Body: <p>for regression, you can use a hidden layer with sigmoid, then a LINEAR output layer, where the weighted sum goes straight through, without modification.</p>
  
  <p>this way your output is not restricted to 0-1</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T18:54:35.687Z
Answers.Text: Alternatively, you may use ReLu activation funtion. ReLu stands for Rectified Linear units and is defined as : max(x, 0)
  
  So , when dealing with problem where the output is not restricted just to 0-1, you may want to use "ReLu" activation function for all the hidden layers and "linear" activation function for the output layer.
Answers.UserDisplayName: SAGALPREET SINGH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-10T15:53:43.380Z
Answers.LastActivityDate: 2020-05-10T15:53:43.380Z
Answers.UserDisplayName: nikos
Answers.Body: <p>Usually you're normalizing the data first, meaning that your whole dataset will be in between 0 and 1. Afterwords after you're having the model predictions, when computing the cost function or evaluating the model, you can apply the inverse of the normalization function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T23:35:57.157Z
Answers.Text: I'm a little confused now, the point of the sigmoid function is to put the data set between 0 and 1?
Answers.UserDisplayName: KovacsMarton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T08:13:17.793Z
Answers.Text: No, you use sigmoid to introduce non-linearity in your model. In addition sigmoid is especially used for models where we have to predict the probability as an output since it's defined in the range of 0 and 1. For normalizing the data i would suggest you to look at min-max normalization or z-score normalization.
Answers.UserDisplayName: razvanc92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T21:43:32.357Z
Answers.LastActivityDate: 2019-12-11T21:43:32.357Z
Answers.UserDisplayName: razvanc92
Body: <p>I'd like to ask for any kind of assistance regarding the following problem:</p>
  
  <p>I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given).This is one instance,I have 20 000 instances for training.Next, I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.</p>
  
  <p>I am stuck because I only know of the sigmoid activation function so far, and I assume it's not suitable for cases like this where the output values aren't either 0 or 1.</p>
  
  <p>So my question is this : What's a good choice for an activation function and how does one go about implementing a neural network for a problem such as this?</p>
  
Tags: <neural-networks><deep-learning><backpropagation><regression>
Title: Regression using neural network
UserDisplayName: KovacsMarton
ThreadText: # Regression using neural network
  
  (Asked by: username_0 on 2019-12-11)
  
  I'd like to ask for any kind of assistance regarding the following problem:
  
  I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given).This is one instance,I have 20 000 instances for training.Next, I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.
  
  I am stuck because I only know of the sigmoid activation function so far, and I assume it's not suitable for cases like this where the output values aren't either 0 or 1.
  
  So my question is this : What's a good choice for an activation function and how does one go about implementing a neural network for a problem such as this?
  
  **Tags:** neural-networks, deep-learning, backpropagation, regression
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2020-05-11. Score: 2
  
  Lets mock some data up.
  
  "100 numbers, each one is a parameter, they together define a number X(also given)"
  
  # i.e. size of X_train -> [n x d]
  # i.e. size of X_train -> [??? x 100]  , when d = 100
  
  # "I have 20000 instances for training"
  # i.e. size of X_train -> [20000 x 100], when n = 20000
  
  import torch
  import numpy as np
  
  X_train = torch.rand((20000, 100))
  X_train = np.random.rand(20000, 100) # Or using numpy
  
  But what is your Y?
  # Since the definition of a regression task, 
  # loosely means to predict an output real number
  # given an input of d dimension
  
  # So the appropriate Y_train would 
  # be of dimension [n x 1] 
  # and look like this:
  
  y_train = torch.rand((20000, 1))
  
  y_train = np.random.rand(20000, 1) # Or using numpy
  
  What is a linear perceptron?
  Taking definition from this tutorial
  
  Thus, in picture:
  
  Next we need to define training routine,
  For now take it as biblical truth that this is an okay routine to train a neural net model (this isn't the only way but easiest or supervised learning):
  
  In code:
  import math
  import numpy as np
  np.random.seed(0)
  
  def sigmoid(x): # Returns values that sums to one.
      return 1 / (1 + np.exp(-x))
  
  def sigmoid_derivative(sx): 
      # See https://math.stackexchange.com/a/1225116
      # Hint: let sx = sigmoid(x)
      return sx * (1 - sx)
  
  def cost(predicted, truth):
      return np.abs(truth - predicted)
  
  
  num_epochs = 10000 # No. of times to iterate.
  learning_rate = 0.03 # How large a step to take per iteration.
  
  # Lets standardize and call our inputs X and outputs Y
  X = np.array(torch.rand((20000, 100)))
  Y = or_output
  
  for _ in range(num_epochs):
      layer0 = X
  
      # Step 2a: Multiply the weights vector with the inputs, sum the products, i.e. s
      # Step 2b: Put the sum through the sigmoid, i.e. f()
      # Inside the perceptron, Step 2. 
      layer1 = sigmoid(np.dot(X, W))
  
      # Back propagation.
      # Step 3a: Compute the errors, i.e. difference between expected output and predictions
      # How much did we miss?
      layer1_error = cost(layer1, Y)
  
      # Step 3b: Multiply the error with the derivatives to get the delta
      # multiply how much we missed by the slope of the sigmoid at the values in layer1
      layer1_delta = layer1_error * sigmoid_derivative(layer1)
  
      # Step 3c: Multiply the delta vector with the inputs, sum the product (use np.dot)
      # Step 4: Multiply the learning rate with the output of Step 3c.
      W +=  learning_rate * np.dot(layer0.T, layer1_delta)
  
  Now that we learn the model, i.e. the W. 
  When we see the data points that we need to use the model on, we apply the same forward propagation step, i.e. layer1 = sigmoid(np.dot(X, W)) 
  Since we have:
  
  I have 5000 lines given, each containing the 100 numbers as parameters.My task is to predict the number X for these 5000 instances.
  
  And in code:
  # If we mock up the data,
  # it should be the same internal dimension. 
  X_test = np.random.rand(5000, 100)
  
  # The desired output just needs to pass through the W and the activation:
  # the shape of `output` -> [5000 x 1] , 
  # where there's 1 output value for each input.
  output = sigmoid(np.dot(X_test, W))
  
  > **Comments:**
  > **username_1**: There's some weird things happening here when we design our network with a sigmoid, we naturally force the output values to be between the `(0, 1)` range. But this is partly because when we mock up the data using `np.random.rand`, the values are expected to fall between the `(0,1)`.  
  > **username_1**: A normal regression task where the output has arbitrary range of values, simply skip the sigmoid layer for the forward propagation and then the error derivatives in the `sigmoid_derivative(layer1)` would just be 1.  
  
  ---
  
  ### Answer by username_2 on 2020-05-11. Score: 2
  
  The quick answer is that you want to use an activation function on the output layer that does not compress values to $(0,1)$. Depending on your software, this might be called "linear" or "identity". It looks like Keras just wants you to leave off the activation function: `model.add(Dense(1))`.
  
  The typical way of thinking of a neural network as a classifier (let's say a binary classifier) is just extending a logistic regression. In fact, when you use a sigmoid activation function on the output node, you're (sort of) running logistic regression on the final hidden layer.
  
  A logistic regression is one type of generalized linear model. The gist of GLM is that some transformation of the the value of interest is a linear function of the feature space.
  
  Let $X$ be the data matrix for the feature space. Let $\beta$ be a parameter vector. Then $\hat{y} = \mathbb{E}[y] = X\beta$ is the linear model, and $g(\mathbb{E}[y]) = X\beta$ is the generalized linear model (vectorized, so apply $g$ to each $y\_i$).
  
  But we could extend this to a nonlinear transformation, and when a neural network is a binary classifier, this is precisely what we're doing. Instead of the transformation of $X$ being given by $\beta$ and thus linear, we apply some nonlinear transformation $f$ and get $g(\mathbb{E}[y]) = f(X)$.
  
  The terminology in GLM is "link function", but that is essentially the activation function on the final node(s) of the neural network. Consequently, all of the GLM link functions are in play, and one of those link functions is the identity function. For a GLM, that's just linear regression. For your neural network, it will be a neural network (nonlinear) regression, which sounds like what you want.
  
  
  ---
  
  ### Answer by username_3 on 2020-05-10. Score: 1
  
  for regression, you can use a hidden layer with sigmoid, then a LINEAR output layer, where the weighted sum goes straight through, without modification.
  
  this way your output is not restricted to 0-1
  
  > **Comments:**
  > **username_4**: Alternatively, you may use ReLu activation funtion. ReLu stands for Rectified Linear units and is defined as : max(x, 0)  So , when dealing with problem where the output is not restricted just to 0-1, you may want to use "ReLu" activation function for all the hidden layers and "linear" activation function for the output layer.  
  
  ---
  
  ### Answer by username_5 on 2019-12-11. Score: 1
  
  Usually you're normalizing the data first, meaning that your whole dataset will be in between 0 and 1. Afterwords after you're having the model predictions, when computing the cost function or evaluating the model, you can apply the inverse of the normalization function.
  
  > **Comments:**
  > **username_0**: I'm a little confused now, the point of the sigmoid function is to put the data set between 0 and 1?  
  > **username_5**: No, you use sigmoid to introduce non-linearity in your model. In addition sigmoid is especially used for models where we have to predict the probability as an output since it's defined in the range of 0 and 1. For normalizing the data i would suggest you to look at min-max normalization or z-score normalization.  
  
  ---
  
  