Answers.Body: <p>It depends on the design of the model, of course.</p>
  <p>The overall design of a GPT model is like this: the last certain number of tokens (e.g. 2048) are passed into a neural network which outputs probabilities for the next token. Then a next token is chosen according to the probabilities. This repeats until something. So it can't see more than a certain number of tokens back - and that's the fault of how it's designed. If you ask it to generate a longer palindrome, it will eventually forget the beginning of the palindrome and also forget the instruction to generate a palindrome. However, if you ask it to generate random numbers, as long as the inputs are random numbers it will probably keep generating more random numbers.</p>
  <p>Other kinds of models exist, like RNNs, which process one token at a time and remember some data between tokens. Theoretically, an RNN could keep passing information down the chain indefinitely. In practice, they do not. Also, nobody has made one work as well as GPT.</p>
  <p>Transformer models, the precursor to GPT, worked like GPT but they did have separate prompt and output strings. The input is the last certain number of output tokens, plus all the input tokens. This kind of model would not forget the instruction to generate a palindrome, but it would forget the beginning of the palindrome after a while.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-03T11:09:43.613Z
Answers.Text: An encoder-decoder transformer isn't a precursor to GPT; it is for machine translation, whilst GPT, the transformer decoder, is for causal language modelling. Also, there is nothing in the design of transformers or GPT that fixes the input size (which is how recent GPT models are able to have very long contexts); they are as recurrent as an RNN, except with better long-range dependencies since there is no bottleneck that moves through the context.
Answers.UserDisplayName: Mew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-02T17:24:06.340Z
Answers.LastActivityDate: 2023-04-02T17:24:06.340Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>Lets assume two types of prompt:
  A fixed prompt for which reasonable responses can be infinite. For example:</p>
  <pre><code>
  &gt; output a random number
  &gt; output a palindrome
  </code></pre>
  <p>A prompt that can slightly varied to get different output each time which reasonable mapping. Quotes mark what will be varied.</p>
  <pre><code>&gt; what is the double of &quot;2&quot;
  &gt; echo &quot;xyz&quot;
  </code></pre>
  <p>Now that the network is trained on finite data and has finite parameters, does it inherently   prevent it from perfectly answering these questions each time.</p>
  <p>If yes, can a tiny network be handcrafted to model an infinite domain function and its only the &quot;learning from data&quot; part that prevents it from learning such functions.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-03T13:15:19.493Z
Comments.Text: ChatGPT cannot even count the words in sentence.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-neural-networks><chatgpt>
Title: Can models like chatGPT learn functions with infinite domain or range
UserDisplayName: Sayam Qazi
ThreadText: # Can models like chatGPT learn functions with infinite domain or range
  
  (Asked by: username_0 on 2023-04-02)
  
  Lets assume two types of prompt:
  A fixed prompt for which reasonable responses can be infinite. For example:
  
  ```
  
  > output a random number
  > output a palindrome
  
  ```
  
  A prompt that can slightly varied to get different output each time which reasonable mapping. Quotes mark what will be varied.
  
  ```
  > what is the double of "2"
  > echo "xyz"
  
  ```
  
  Now that the network is trained on finite data and has finite parameters, does it inherently prevent it from perfectly answering these questions each time.
  
  If yes, can a tiny network be handcrafted to model an infinite domain function and its only the "learning from data" part that prevents it from learning such functions.
  
  **Tags:** neural-networks, deep-neural-networks, chatgpt
  
  > **Comments:**
  > **username_1**: ChatGPT cannot even count the words in sentence.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-02. Score: 2 | Accepted answer
  
  It depends on the design of the model, of course.
  
  The overall design of a GPT model is like this: the last certain number of tokens (e.g. 2048) are passed into a neural network which outputs probabilities for the next token. Then a next token is chosen according to the probabilities. This repeats until something. So it can't see more than a certain number of tokens back - and that's the fault of how it's designed. If you ask it to generate a longer palindrome, it will eventually forget the beginning of the palindrome and also forget the instruction to generate a palindrome. However, if you ask it to generate random numbers, as long as the inputs are random numbers it will probably keep generating more random numbers.
  
  Other kinds of models exist, like RNNs, which process one token at a time and remember some data between tokens. Theoretically, an RNN could keep passing information down the chain indefinitely. In practice, they do not. Also, nobody has made one work as well as GPT.
  
  Transformer models, the precursor to GPT, worked like GPT but they did have separate prompt and output strings. The input is the last certain number of output tokens, plus all the input tokens. This kind of model would not forget the instruction to generate a palindrome, but it would forget the beginning of the palindrome after a while.
  
  > **Comments:**
  > **username_3**: An encoder-decoder transformer isn't a precursor to GPT; it is for machine translation, whilst GPT, the transformer decoder, is for causal language modelling. Also, there is nothing in the design of transformers or GPT that fixes the input size (which is how recent GPT models are able to have very long contexts); they are as recurrent as an RNN, except with better long-range dependencies since there is no bottleneck that moves through the context.  
  
  ---
  
  