Answers.Body: <p>To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:</p>
  <ul>
  <li>the odd numbers</li>
  <li>the primes</li>
  <li>numbers expressed in digits, but listed in alphabetical order of their name in German</li>
  <li>numbers listed in the lexical order of the reverse of their representation in base 3</li>
  <li>the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-22T08:25:12.393Z
Answers.Text: Would a https://en.wikipedia.org/wiki/Mixture_of_experts be an appropriate tool to deal with this scenario?
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-23T10:47:38.493Z
Answers.Text: No.
  
  The set of all sequences of whole numbers is uncountably infinite. Even the set of computable sequences of numbers is large and hard to deal with.
  
  Like any other machine learning problem, making progress depends on prior knowledge. 
  
  Here is a subset of the problem which is hard but not impossible, and which relates to some work by Kolmogorov: a sequence of numbers is output by a program, and short programs are considered more likely than long programs. Given the first N terms, where N is large compared to the expected length of the program, predict the N+1 th term.
Answers.UserDisplayName: chrishmorris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-23T17:38:37.540Z
Answers.Text: The odd numbers are very easy: x(n)=2n+1. If we use x(n)=x(n-1)+2, that includes the even numbers as well. If we use x(n)=2x(n-1)-x(n-2), that is all arithmetic sequences.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-25T07:22:27.890Z
Answers.Text: Yes, but this sequence is not easy: 3,5,7. Is that odd numbers>1? Or is it primes>2? Or the start of 3,5,7,3,5,7,3,5,7, ....? Or any other of an infinite number of computable sequences with that prefix?
Answers.UserDisplayName: chrishmorris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-25T11:24:40.607Z
Answers.Text: The simplest model is for odd numbers >1, linear recurrence of order 2. With order 3 you can get 3,5,7,3,5,7... x(n)=x(n-3). No linear model for the primes.
  
  
  An example that makes me crazier is 1,2,... is the simplest interpretation the counting numbers or the powers or 2? Every kid starts by associating that start with counting numbers x(n)=2x(n-1)-x(n-2), but the order of the linear recurrence for the powers of 2 is smaller: x(n)=2x(n-1)
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T15:59:18.860Z
Answers.LastActivityDate: 2021-03-29T15:59:18.860Z
Answers.UserDisplayName: chrishmorris
Body: <p>Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.</p>
  <p>Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-22T07:50:23.030Z
Comments.Text: Are you asking whether the architecture of GPT-2 could be trained from scratch to perform the task of predicting the next number or are you asking whether the trained GPT-2 model could perform number prediction zero-shot (without any fine-tuning)?
Comments.UserDisplayName: Mariusmarten
Tags: <transformer><attention><bert><gpt><forecasting>
Title: Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?
UserDisplayName: Nyxynyx
ThreadText: # Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?
  
  (Asked by: username_0 on 2021-03-28)
  
  Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.
  
  Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?
  
  **Tags:** transformer, attention, bert, gpt, forecasting
  > **Comments:**
  > **username_1**: Are you asking whether the architecture of GPT-2 could be trained from scratch to perform the task of predicting the next number or are you asking whether the trained GPT-2 model could perform number prediction zero-shot (without any fine-tuning)?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-29. Score: 0
  
  To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:
  
  * the odd numbers
  * the primes
  * numbers expressed in digits, but listed in alphabetical order of their name in German
  * numbers listed in the lexical order of the reverse of their representation in base 3
  * the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber
  
  > **Comments:**
  > **username_3**: Would a https://en.wikipedia.org/wiki/Mixture_of_experts be an appropriate tool to deal with this scenario?  
  > **username_2**: No.  The set of all sequences of whole numbers is uncountably infinite. Even the set of computable sequences of numbers is large and hard to deal with.  Like any other machine learning problem, making progress depends on prior knowledge.   Here is a subset of the problem which is hard but not impossible, and which relates to some work by Kolmogorov: a sequence of numbers is output by a program, and short programs are considered more likely than long programs. Given the first N terms, where N is large compared to the expected length of the program, predict the N+1 th term.  
  > **username_3**: The odd numbers are very easy: x(n)=2n+1. If we use x(n)=x(n-1)+2, that includes the even numbers as well. If we use x(n)=2x(n-1)-x(n-2), that is all arithmetic sequences.  
  > **username_2**: Yes, but this sequence is not easy: 3,5,7. Is that odd numbers>1? Or is it primes>2? Or the start of 3,5,7,3,5,7,3,5,7, ....? Or any other of an infinite number of computable sequences with that prefix?  
  > **username_3**: The simplest model is for odd numbers >1, linear recurrence of order 2. With order 3 you can get 3,5,7,3,5,7... x(n)=x(n-3). No linear model for the primes.   An example that makes me crazier is 1,2,... is the simplest interpretation the counting numbers or the powers or 2? Every kid starts by associating that start with counting numbers x(n)=2x(n-1)-x(n-2), but the order of the linear recurrence for the powers of 2 is smaller: x(n)=2x(n-1)  
  
  ---
  
  