Answers.Body: <p>I think your code works fine for what is meant to be doing - fitting a linear regression model. The problem here is that you are using a linear model. Linear model does not have an adequate approximation capacity, it will only be able to fit data that is described by a linear function. Here, you gave a random sequence of numbers, that is very difficult for linear model to approximate. I would advise you to try 2 things:  </p>
  
  <p>1) Try something simpler first. Instead of doing a random sequence of numbers do a linear sequence of numbers for example a function like <span class="math-container">$y = 2x$</span> or maybe affine function like <span class="math-container">$y = 2x + 5$</span>. So you would have a sequence like:  </p>
  
  <p><span class="math-container">$2, 4, 6, 8 ...$</span> or <span class="math-container">$7, 9, 11, 13, ...$</span>  </p>
  
  <p>If you manage to get that working try a nonlinear function like <span class="math-container">$x^2$</span> for example.  </p>
  
  <p>2) Instead of using a linear model try a nonlinear model. For example a polynomial regression model. Especially powerful function approximators are neural networks. In theory, a neural network with single hidden layer can approximate an arbitrary continuous function under some conditions 
  (<a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">Universal approximation theorem</a>) , so you could try to see how would a neural network solve the problem, there are several open source neural network libraries that you could try.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-05T11:46:53.170Z
Answers.Text: well I found a pattern and it goes like this: we take the first element in the array and then we take it's index and next index and then the AI makes the decision which number to choose in that interval, that's how we got the number for example we got 3(2nd number in array) by applying the same logic like this [2^1(index of 2nd element), **3**(number 3 exists somewhere between this interval), 2^2(next index)] and so on and so forth for each element (his index and the next...) but since I have no previous knowledge in AI and machine learning I have no idea how to apply this
Answers.UserDisplayName: Thorvald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-27T17:27:30.323Z
Answers.LastActivityDate: 2019-04-27T17:27:30.323Z
Answers.UserDisplayName: Brale
Answers.Body: <p>Like others said you can't approximate this with a linear regression model.<br><br>
  A PRM that approximates a solution could give you the following:<br>
  <span class="math-container">$y = 0.948 + x + 0.00085*x^6$</span> ~ <br>
  <span class="math-container">$y = 237/250 + x + (17/20000)*x^6$</span><br>
  For <span class="math-container">$x = 9$</span>, <span class="math-container">$y \simeq 462$</span><br></p>
  
  <p>or </p>
  
  <p><span class="math-container">$y = 0.9258 + x + 0.00086*x^6$</span>
  For <span class="math-container">$x = 9$</span>, <span class="math-container">$y \simeq 466.965$</span><br></p>
  
  <p><code>UPDATE</code><br>
  An approximation of course, may be in the range of:<br>
  <span class="math-container">$y = 2^{(x + 1)} - 2^x$</span> -the model you propose- <br>
  <code>Goodness of fit: 0.968475 and Mean Square Error = 685.111</code><br><br>
  Based on this range a better approximation would be:<br>
  <span class="math-container">$y = 2^x + (-1/2)*x^2$</span><br>
  with <span class="math-container">$R^2$</span><br>
  <code>Goodness of fit = 0.995</code> <br>
  <code>Mean Square Error: 89.0278</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-09T10:14:18.820Z
Answers.Text: I actually found a pattern it is not random the pattern goes like this: you take the element's current index and it's next index and you apply this interval : [2^index, 2^next index] for example for the number 7 in the given example above we do this [2^2, 2^3] the number 7 is somewhere between that interval, the AI must predict based on this pattern. if you have any solution proposed please be kind and guide me through!
Answers.UserDisplayName: Thorvald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-09T09:34:11.717Z
Answers.LastActivityDate: 2019-05-09T10:43:17.573Z
Answers.LastEditDate: 2019-05-09T10:43:17.573Z
Answers.UserDisplayName: Panos Kalatzantonakis
Body: <p>The AI must predict the next number in a given sequence of incremental integers (with no obvious pattern) using Python but so far I don't get the intended result!
  I tried changing the learning rate and iterations but so far no luck!</p>
  
  <p>Example sequence: [1, 3, 7, 8, 21, 49, 76, 224]</p>
  
  <p>Expected result: 467</p>
  
  <p>Result found : 2,795.5</p>
  
  <p>Cost: 504579.43</p>
  
  <p>This is what I've done so far:</p>
  
  <pre><code>import numpy as np
  
  # Init sequence
  data =\
      [
          [0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
          [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]
      ]
  
  X = np.matrix(data)[:, 0]
  y = np.matrix(data)[:, 1]
  
  def J(X, y, theta):
      theta = np.matrix(theta).T
      m = len(y)
      predictions = X * theta
      sqError = np.power((predictions-y), [2])
      return 1/(2*m) * sum(sqError)
  
  dataX = np.matrix(data)[:, 0:1]
  X = np.ones((len(dataX), 2))
  X[:, 1:] = dataX
  
  # gradient descent function
  def gradient(X, y, alpha, theta, iters):
      J_history = np.zeros(iters)
      m = len(y)
      theta = np.matrix(theta).T
      for i in range(iters):
          h0 = X * theta
          delta = (1 / m) * (X.T * h0 - X.T * y)
          theta = theta - alpha * delta
          J_history[i] = J(X, y, theta.T)
       return J_history, theta
  print('\n'+40*'=')
  
  # Theta initialization
  theta = np.matrix([np.random.random(), np.random.random()])
  
  # Learning rate
  alpha = 0.02
  
  # Iterations
  iters = 1000000
  
  print('\n== Model summary ==\nLearning rate: {}\nIterations: {}\nInitial 
  theta: {}\nInitial J: {:.2f}\n'
    .format(alpha, iters, theta, J(X, y, theta).item()))
  print('Training model... ')
  
  # Train model and find optimal Theta value
  J_history, theta_min = gradient(X, y, alpha, theta, iters)
  print('Done, Model is trained')
  print('\nModelled prediction function is:\ny = {:.2f} * x + {:.2f}'
    .format(theta_min[1].item(), theta_min[0].item()))
  print('Cost is: {:.2f}'.format(J(X, y, theta_min.T).item()))
  
  # Calculate the predicted profit
  def predict(pop):
      return [1, pop] * theta_min
  
  # Now
  p = len(data)
  print('\n'+40*'=')
  print('Initial sequence was:\n', *np.array(data)[:, 1])
  print('\nNext numbers should be: {:,.1f}'
    .format(predict(p).item()))
  </code></pre>
  
  <p><strong>UPDATE</strong> Another method I tried but still giving wrong results</p>
  
  <pre><code>import numpy as np
  from sklearn import datasets, linear_model
  
  # Define the problem
  problem = [1, 3, 7, 8, 21, 49, 76, 224]
  
  # create x and y for the problem
  
  x = []
  y = []
  
  for (xi, yi) in enumerate(problem):
      x.append([xi])
      y.append(yi)
  
  x = np.array(x)
  y = np.array(y)
  # Create linear regression object
  regr = linear_model.LinearRegression()
  regr.fit(x, y)
  
  # create the testing set
  x_test = [[i] for i in range(len(x), 3 + len(x))]
  
  # The coefficients
  print('Coefficients: \n', regr.coef_)
  # The mean squared error
  print("Mean squared error: %.2f" % np.mean((regr.predict(x) - y) ** 2))
  # Explained variance score: 1 is perfect prediction
  print('Variance score: %.2f' % regr.score(x, y))
  
  # Do predictions
  y_predicted = regr.predict(x_test)
  
  print("Next few numbers in the series are")
  for pred in y_predicted:
      print(pred)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-28T09:54:04.687Z
Comments.Text: Cross-posted here: https://datascience.stackexchange.com/q/51033/924
Comments.UserDisplayName: Has QUIT--Anony-Mousse
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-28T11:15:28.727Z
Comments.Text: are you hunting my posts? @Anony-Mousse, I posted the same thread on multiple Stack platforms
Comments.UserDisplayName: Thorvald
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-29T01:31:16.367Z
Comments.Text: @nbro please be clear to suggest to only ask in *one* place, and not to duplicate questions. Otherwise future users will have to look for answers in many places. For the rationale of not allowing crossposting see https://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu/64069#64069 Thank you.
Comments.UserDisplayName: Has QUIT--Anony-Mousse
Tags: <neural-networks><machine-learning><training><python>
Title: Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers
UserDisplayName: Thorvald
ThreadText: # Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers
  
  (Asked by: username_0 on 2019-04-27)
  
  The AI must predict the next number in a given sequence of incremental integers (with no obvious pattern) using Python but so far I don't get the intended result!
  I tried changing the learning rate and iterations but so far no luck!
  Example sequence: [1, 3, 7, 8, 21, 49, 76, 224]
  Expected result: 467
  Result found : 2,795.5
  Cost: 504579.43
  This is what I've done so far:
  import numpy as np
  
  # Init sequence
  data =\
      [
          [0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
          [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]
      ]
  
  X = np.matrix(data)[:, 0]
  y = np.matrix(data)[:, 1]
  
  def J(X, y, theta):
      theta = np.matrix(theta).T
      m = len(y)
      predictions = X * theta
      sqError = np.power((predictions-y), [2])
      return 1/(2*m) * sum(sqError)
  
  dataX = np.matrix(data)[:, 0:1]
  X = np.ones((len(dataX), 2))
  X[:, 1:] = dataX
  
  # gradient descent function
  def gradient(X, y, alpha, theta, iters):
      J_history = np.zeros(iters)
      m = len(y)
      theta = np.matrix(theta).T
      for i in range(iters):
          h0 = X * theta
          delta = (1 / m) * (X.T * h0 - X.T * y)
          theta = theta - alpha * delta
          J_history[i] = J(X, y, theta.T)
       return J_history, theta
  print('\n'+40*'=')
  
  # Theta initialization
  theta = np.matrix([np.random.random(), np.random.random()])
  
  # Learning rate
  alpha = 0.02
  
  # Iterations
  iters = 1000000
  
  print('\n== Model summary ==\nLearning rate: {}\nIterations: {}\nInitial 
  theta: {}\nInitial J: {:.2f}\n'
    .format(alpha, iters, theta, J(X, y, theta).item()))
  print('Training model... ')
  
  # Train model and find optimal Theta value
  J_history, theta_min = gradient(X, y, alpha, theta, iters)
  print('Done, Model is trained')
  print('\nModelled prediction function is:\ny = {:.2f} * x + {:.2f}'
    .format(theta_min[1].item(), theta_min[0].item()))
  print('Cost is: {:.2f}'.format(J(X, y, theta_min.T).item()))
  
  # Calculate the predicted profit
  def predict(pop):
      return [1, pop] * theta_min
  
  # Now
  p = len(data)
  print('\n'+40*'=')
  print('Initial sequence was:\n', *np.array(data)[:, 1])
  print('\nNext numbers should be: {:,.1f}'
    .format(predict(p).item()))
  
  UPDATE Another method I tried but still giving wrong results
  import numpy as np
  from sklearn import datasets, linear_model
  
  # Define the problem
  problem = [1, 3, 7, 8, 21, 49, 76, 224]
  
  # create x and y for the problem
  
  x = []
  y = []
  
  for (xi, yi) in enumerate(problem):
      x.append([xi])
      y.append(yi)
  
  x = np.array(x)
  y = np.array(y)
  # Create linear regression object
  regr = linear_model.LinearRegression()
  regr.fit(x, y)
  
  # create the testing set
  x_test = [[i] for i in range(len(x), 3 + len(x))]
  
  # The coefficients
  print('Coefficients: \n', regr.coef_)
  # The mean squared error
  print("Mean squared error: %.2f" % np.mean((regr.predict(x) - y) ** 2))
  # Explained variance score: 1 is perfect prediction
  print('Variance score: %.2f' % regr.score(x, y))
  
  # Do predictions
  y_predicted = regr.predict(x_test)
  
  print("Next few numbers in the series are")
  for pred in y_predicted:
      print(pred)
  
  **Tags:** neural-networks, machine-learning, training, python
  > **Comments:**
  > **username_1**: Cross-posted here: https://datascience.stackexchange.com/q/51033/924  
  > **username_0**: are you hunting my posts? @Anony-Mousse, I posted the same thread on multiple Stack platforms  
  > **username_1**: @nbro please be clear to suggest to only ask in *one* place, and not to duplicate questions. Otherwise future users will have to look for answers in many places. For the rationale of not allowing crossposting see https://meta.stackexchange.com/questions/64068/is-cross-posting-a-question-on-multiple-stack-exchange-sites-permitted-if-the-qu/64069#64069 Thank you.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-04-27. Score: 3
  
  I think your code works fine for what is meant to be doing - fitting a linear regression model. The problem here is that you are using a linear model. Linear model does not have an adequate approximation capacity, it will only be able to fit data that is described by a linear function. Here, you gave a random sequence of numbers, that is very difficult for linear model to approximate. I would advise you to try 2 things:
  
  1) Try something simpler first. Instead of doing a random sequence of numbers do a linear sequence of numbers for example a function like $y = 2x$ or maybe affine function like $y = 2x + 5$. So you would have a sequence like:
  
  $2, 4, 6, 8 ...$ or $7, 9, 11, 13, ...$
  
  If you manage to get that working try a nonlinear function like $x^2$ for example.
  
  2) Instead of using a linear model try a nonlinear model. For example a polynomial regression model. Especially powerful function approximators are neural networks. In theory, a neural network with single hidden layer can approximate an arbitrary continuous function under some conditions
  ([Universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem)) , so you could try to see how would a neural network solve the problem, there are several open source neural network libraries that you could try.
  
  > **Comments:**
  > **username_0**: well I found a pattern and it goes like this: we take the first element in the array and then we take it's index and next index and then the AI makes the decision which number to choose in that interval, that's how we got the number for example we got 3(2nd number in array) by applying the same logic like this [2^1(index of 2nd element), **3**(number 3 exists somewhere between this interval), 2^2(next index)] and so on and so forth for each element (his index and the next...) but since I have no previous knowledge in AI and machine learning I have no idea how to apply this  
  
  ---
  
  ### Answer by username_3 on 2019-05-09. Score: 2
  
  Like others said you can't approximate this with a linear regression model.  
    
  A PRM that approximates a solution could give you the following:  
  $y = 0.948 + x + 0.00085\*x^6$ ~   
  $y = 237/250 + x + (17/20000)\*x^6$  
  For $x = 9$, $y \simeq 462$
  
  or
  
  $y = 0.9258 + x + 0.00086\*x^6$
  For $x = 9$, $y \simeq 466.965$
  
  `UPDATE`  
  An approximation of course, may be in the range of:  
  $y = 2^{(x + 1)} - 2^x$ -the model you propose-   
  `Goodness of fit: 0.968475 and Mean Square Error = 685.111`  
    
  Based on this range a better approximation would be:  
  $y = 2^x + (-1/2)\*x^2$  
  with $R^2$  
  `Goodness of fit = 0.995`   
  `Mean Square Error: 89.0278`
  
  > **Comments:**
  > **username_0**: I actually found a pattern it is not random the pattern goes like this: you take the element's current index and it's next index and you apply this interval : [2^index, 2^next index] for example for the number 7 in the given example above we do this [2^2, 2^3] the number 7 is somewhere between that interval, the AI must predict based on this pattern. if you have any solution proposed please be kind and guide me through!  
  
  ---
  
  