Answers.Body: <p>They have no such guarantee. This is true for many reasons, but the simplest of which is:</p>
  <ol>
  <li>The definition of correct output is very underspecified.</li>
  </ol>
  <p>We do not have a rigorous way of defining exactly what sort of program is being defined by the natural language, so it would make no sense to think an AI could rigorously be proven to fulfill that requirement.</p>
  <p>While that is itself sufficient, there is one more fundamental reason why there is no such guarantee:</p>
  <ol start="2">
  <li>While it depends on the code generation solution, many are based on GPTs or similar types of architectures that are probabilistic and prone to hallucination. In fact, any sort of Machine Learning based solution will defy such proofs. At best we can make a type of <a href="https://en.wikipedia.org/wiki/Probably_approximately_correct_learning#:%7E:text=In%20computational%20learning%20theory%2C%20probably,in%201984%20by%20Leslie%20Valiant." rel="noreferrer">PAC</a> argument</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-26T01:37:20.193Z
Answers.Text: You don't need to make a PAC argument, since we actually have test sets available. PAC guarantees tell you that if your function class is simple enough, your test error will be within some interval of your train error. In practice this doesn't matter at all since we can't calculate the bounds, and our function classes aren't simple.
  
  Fortunately it doesn't matter: If we have some representative set of tasks, and the model is successful 80% of the time, we can just use statistics 101 stuff to get a bound on its error rate.
Answers.UserDisplayName: Davis Yoshida
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-26T12:56:28.590Z
Answers.Text: Even when you think it is correctly specified, when you have a multi-step process of additions and modifications to the original code it's understandably hard for AI to 'remember' or even understand what context is still relevant and what is not. Hence the output being unexpectedly wrong at times.
Answers.UserDisplayName: Mast
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-25T17:44:33.180Z
Answers.LastActivityDate: 2023-12-25T17:44:33.180Z
Answers.UserDisplayName: chessprogrammer
Answers.Body: <blockquote>
  <p>Do AI based code generators guarantee correct output?</p>
  </blockquote>
  <p>No. There is no written guarantee by any provider.</p>
  <p>Researchers have measured &quot;accuracy&quot; (there's more than one way to think about this of course, since code may solve a problem, but be poorly written), and they do not find it is 100% accurate. <a href="https://akvelon.com/github-copilot-efficiency-survey-data-revealed-akvelon/#:%7E:text=Based%20on%20the%20findings%20from,the%20tool%20may%20be%20helpful." rel="nofollow noreferrer">A value of 80% accuracy was suggested in a survey by Akevlon</a>.</p>
  <blockquote>
  <p>What mechanism is used to generate code and how is it different than standard NLP?</p>
  </blockquote>
  <p>It is the same models as NLP, except for data it has been trained on (although LLMs like ChatGPT are general enough that the exact same model - trained on code, poetry, newspaper articles, novels etc - performs coding alongside natural language tasks). This may be augmented at generation phase, because it is possible to use compilers to filter out things that don't work at all. But compilers cannot determine that a task is performed correctly or that suitable names are used for variables etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-25T20:32:09.607Z
Answers.Text: Is it guaranteed to even be syntactically correct? As in generate a compilable program? A while back I saw a research presentation where basically they tried to mix RL, ML and classic NLP(tokens and shit) to guarantee a compileable program.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-25T20:45:38.733Z
Answers.Text: @FourierFlux - in general no, but as I point out in the answer, the makers can in theory use a compiler to filter against multiple generations and pick one with least errors. I don't know if anyone does that, but I suspect some code gen services will because it is a lot easier than deploying the AI side of the model
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-25T17:55:28.150Z
Answers.LastActivityDate: 2023-12-25T17:55:28.150Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Current implementations do not.</p>
  <p>However, if you had an exact and executable definition of &quot;correct output&quot;, such as a comprehensive test suite, it could be combined with the code generator. Most language models are quite adept at improving the code based on feedback, so for most assignments this would lead to code that fulfills all test cases.</p>
  <p>There is still a risk of over-specialization, leading to code that works correctly for the test cases, but not for inputs outside that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-26T08:41:21.993Z
Answers.LastActivityDate: 2023-12-26T08:41:21.993Z
Answers.UserDisplayName: jpa
Body: <p>I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-27T16:10:30.987Z
Comments.Text: Human coders can't guarantee correct code, and we don't have any way to check whether code is correct (except in some trivial cases). How could we possibly expect more of AI coders?
Comments.UserDisplayName: Barmar
Tags: <natural-language-processing><chatgpt><large-language-models>
Title: Do AI-based code-generators guarantee correct output?
UserDisplayName: FourierFlux
ThreadText: # Do AI-based code-generators guarantee correct output?
  
  (Asked by: username_0 on 2023-12-25)
  
  I haven't spent much time looking at AI-based code-generators. What mechanism is used to generate code and how is it different than standard NLP?
  
  **Tags:** natural-language-processing, chatgpt, large-language-models
  
  > **Comments:**
  > **username_1**: Human coders can't guarantee correct code, and we don't have any way to check whether code is correct (except in some trivial cases). How could we possibly expect more of AI coders?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-12-25. Score: 10
  
  They have no such guarantee. This is true for many reasons, but the simplest of which is:
  
  1. The definition of correct output is very underspecified.
  
  We do not have a rigorous way of defining exactly what sort of program is being defined by the natural language, so it would make no sense to think an AI could rigorously be proven to fulfill that requirement.
  
  While that is itself sufficient, there is one more fundamental reason why there is no such guarantee:
  
  2. While it depends on the code generation solution, many are based on GPTs or similar types of architectures that are probabilistic and prone to hallucination. In fact, any sort of Machine Learning based solution will defy such proofs. At best we can make a type of [PAC](https://en.wikipedia.org/wiki/Probably_approximately_correct_learning#:%7E:text=In%20computational%20learning%20theory%2C%20probably,in%201984%20by%20Leslie%20Valiant.) argument
  
  > **Comments:**
  > **username_3**: You don't need to make a PAC argument, since we actually have test sets available. PAC guarantees tell you that if your function class is simple enough, your test error will be within some interval of your train error. In practice this doesn't matter at all since we can't calculate the bounds, and our function classes aren't simple.  Fortunately it doesn't matter: If we have some representative set of tasks, and the model is successful 80% of the time, we can just use statistics 101 stuff to get a bound on its error rate.  
  > **username_4**: Even when you think it is correctly specified, when you have a multi-step process of additions and modifications to the original code it's understandably hard for AI to 'remember' or even understand what context is still relevant and what is not. Hence the output being unexpectedly wrong at times.  
  
  ---
  
  ### Answer by username_5 on 2023-12-25. Score: 3
  
  > Do AI based code generators guarantee correct output?
  
  No. There is no written guarantee by any provider.
  
  Researchers have measured "accuracy" (there's more than one way to think about this of course, since code may solve a problem, but be poorly written), and they do not find it is 100% accurate. [A value of 80% accuracy was suggested in a survey by Akevlon](https://akvelon.com/github-copilot-efficiency-survey-data-revealed-akvelon/#:%7E:text=Based%20on%20the%20findings%20from,the%20tool%20may%20be%20helpful.).
  
  > What mechanism is used to generate code and how is it different than standard NLP?
  
  It is the same models as NLP, except for data it has been trained on (although LLMs like ChatGPT are general enough that the exact same model - trained on code, poetry, newspaper articles, novels etc - performs coding alongside natural language tasks). This may be augmented at generation phase, because it is possible to use compilers to filter out things that don't work at all. But compilers cannot determine that a task is performed correctly or that suitable names are used for variables etc.
  
  > **Comments:**
  > **username_0**: Is it guaranteed to even be syntactically correct? As in generate a compilable program? A while back I saw a research presentation where basically they tried to mix RL, ML and classic NLP(tokens and shit) to guarantee a compileable program.  
  > **username_5**: @username_0 - in general no, but as I point out in the answer, the makers can in theory use a compiler to filter against multiple generations and pick one with least errors. I don't know if anyone does that, but I suspect some code gen services will because it is a lot easier than deploying the AI side of the model  
  
  ---
  
  ### Answer by username_6 on 2023-12-26. Score: 2
  
  Current implementations do not.
  
  However, if you had an exact and executable definition of "correct output", such as a comprehensive test suite, it could be combined with the code generator. Most language models are quite adept at improving the code based on feedback, so for most assignments this would lead to code that fulfills all test cases.
  
  There is still a risk of over-specialization, leading to code that works correctly for the test cases, but not for inputs outside that.
  
  
  ---
  
  