Answers.Body: <p>The so called Turing-barrier is a hard theoretical limit that no system, biological or artificial, can surpass. The so called training-wall observed in the latest large AI systems arises from practical and engineering limitations rather than theoretical barriers, such as diminishing returns, architectural bottlenecks, data exhaustion, energy and cost constraints, and lack of reasoning/abstraction/out-of-distribution universally despite their ability to mimic such behavior in narrow contexts. There's no proof that the way to solve any of these practical limitations is undecidable by humans intrinsically, they're just typical complex social and engineering decision problems facing various kinds of <em>complexity</em> for trade-offs beyond mere theoretic <em>computability</em>.</p>
  <p>While the Turing barrier cannot be broken, the training wall may still be shifted or circumvented, but perhaps not completely. For example, some new smarter training paradigm such as hybrid neuro-symbolic system and meta-learning framework could possibly further break some of the above mentioned training walls in the future. Or some new model beyond Transformer is found to be better suited for logical reasoning along with long-term memory attention, at least in some specialized field.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-26T02:56:32.673Z
Answers.LastActivityDate: 2024-12-26T02:56:32.673Z
Answers.UserDisplayName: cinch
Body: <p>There are serious hints that current deep learning approaches of LLM (or VLA - Visual-Language-Action models generally) may be hitting a wall <a href="https://www.pcgamer.com/software/ai/open-ai-co-founder-reckons-ai-training-has-hit-a-wall-forcing-ai-labs-to-train-their-models-smarter-not-just-bigger/" rel="nofollow noreferrer">https://www.pcgamer.com/software/ai/open-ai-co-founder-reckons-ai-training-has-hit-a-wall-forcing-ai-labs-to-train-their-models-smarter-not-just-bigger/</a> There is also WSJ (paywalled, but very illuminating) article <a href="https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693" rel="nofollow noreferrer">https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693</a> with the leaks from the trenches of OpenAI that suggests that the path forward to the new iteration of LLMs is quite challenging indeed and this article does not project any optimism beyond extension of the applications and the sugars of the current state of the GenAI.</p>
  <p>When we are trying to grasp, what the deep learning is doing, then we may think that deep learning is learning the <em>intelligence function</em> that receives task and outputs skill (solution method) in the approach of &quot;On the Measure of Intelligence&quot; <a href="https://arcprize.org/arc" rel="nofollow noreferrer">https://arcprize.org/arc</a> which outlines the criteria for such function which embodies intelligence.</p>
  <p>We know, that function synthesis (and also their verification of correctness) are incomputable and undecidable problems, they are beyond the Turing barrier. It is the common (scholastic) knowledge that the Turing barreir can not be broken down.</p>
  <p>So - may question is: how the (Gen)AI neural training wall is connected with the Turing barrier and if the later is not breakable, then can we say that AI training wall will not be broken down as well?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T12:45:56.043Z
Comments.Text: I don't think that deep learning is learning the intelligence function. Deep learning models are trained to have a skill. If you have just one skill, does it make you intelligent? In fact, the paper that you mention actually states in the abstract that skill alone is not very useful to measure intelligence.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T12:46:42.680Z
Comments.Text: Can you also explain what you mean by "function synthesis"? Can you provide a reliable reference that states that what you mean by function synthesis is incomputable? What do you mean by Turing barrier? Do you mean anything that is not computable by a Turing machine?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T15:30:10.460Z
Comments.Text: @nbro Referenced "On the Measure of Intelligence" clearly states, that the _skill_ (e.g. trained NN) is function that receives input for specific tasks and returns output. It also says, that the _intelligence_ (e.g. NN training setup, data and algo) is the function that receives the task description and generates _skill_ function in the output. There are different levels of intelligence (effectiveness of skill generation), but this ARC article defines the optimal intelligence function. We can discuss whether LLM training is synthesis of skill function for one task, or is it the ...
Comments.UserDisplayName: TomR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T15:32:09.573Z
Comments.Text: ... the synthesis of the set of skill function for the set of tasks and with the choice operators, that allows the recognition of the task and routing the execution to the right function. Or we can hypothesize, that LLM training can be training for intelligence - where LLM can to skill synthesis on the fly. But always - LLM training is the synthesis of the _skill_ or _intelligence_ function (meta-function).
Comments.UserDisplayName: TomR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T15:35:21.350Z
Comments.Text: Each synthesis of function wrt the constraints/specification (the correctness of task execution for the _skill_ function and the efficiency of skill generation for the _intelligence_ function) involved two steps: 1) generation or the function; 2) proof that the generated function is correct wrt specification. The steps can be fused, if the proof is being done constructively. We can posit different things about generation of the function, but it is clear, that the proof step is incomputable un undecidable as it is the subcase of FOL SAT (of SAT of Second Order Logic or even more so for HOL).
Comments.UserDisplayName: TomR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T15:38:39.250Z
Comments.Text: It is about the proof step alone, but one can conclude from e experience that usually the first step (generation) is even more complex. So - generally yes - LLM training hides function synthesis and hence - LLM training hides the efforts to solve incomputable problem. Of course, many practical applications degenerate to the finite and computable cases and it may be great or even sufficient for most cases of applications. But wall there is. And there are certainly tasks where it can be hitting this wall. Ah, an not to mention the more earthly encounters with NP complete problems.
Comments.UserDisplayName: TomR
Tags: <training><large-language-models><turing-completeness>
Title: Can the Turing barrier be behing the AI (neural) training wall and hence - there is no chance to break the AI wall?
UserDisplayName: TomR
ThreadText: # Can the Turing barrier be behing the AI (neural) training wall and hence - there is no chance to break the AI wall?
  
  (Asked by: username_0 on 2024-12-25)
  
  There are serious hints that current deep learning approaches of LLM (or VLA - Visual-Language-Action models generally) may be hitting a wall <https://www.pcgamer.com/software/ai/open-ai-co-founder-reckons-ai-training-has-hit-a-wall-forcing-ai-labs-to-train-their-models-smarter-not-just-bigger/> There is also WSJ (paywalled, but very illuminating) article <https://www.wsj.com/tech/ai/openai-gpt5-orion-delays-639e7693> with the leaks from the trenches of OpenAI that suggests that the path forward to the new iteration of LLMs is quite challenging indeed and this article does not project any optimism beyond extension of the applications and the sugars of the current state of the GenAI.
  
  When we are trying to grasp, what the deep learning is doing, then we may think that deep learning is learning the *intelligence function* that receives task and outputs skill (solution method) in the approach of "On the Measure of Intelligence" <https://arcprize.org/arc> which outlines the criteria for such function which embodies intelligence.
  
  We know, that function synthesis (and also their verification of correctness) are incomputable and undecidable problems, they are beyond the Turing barrier. It is the common (scholastic) knowledge that the Turing barreir can not be broken down.
  
  So - may question is: how the (Gen)AI neural training wall is connected with the Turing barrier and if the later is not breakable, then can we say that AI training wall will not be broken down as well?
  
  **Tags:** training, large-language-models, turing-completeness
  > **Comments:**
  > **username_1**: I don't think that deep learning is learning the intelligence function. Deep learning models are trained to have a skill. If you have just one skill, does it make you intelligent? In fact, the paper that you mention actually states in the abstract that skill alone is not very useful to measure intelligence.  
  > **username_1**: Can you also explain what you mean by "function synthesis"? Can you provide a reliable reference that states that what you mean by function synthesis is incomputable? What do you mean by Turing barrier? Do you mean anything that is not computable by a Turing machine?  
  > **username_0**: @nbro Referenced "On the Measure of Intelligence" clearly states, that the _skill_ (e.g. trained NN) is function that receives input for specific tasks and returns output. It also says, that the _intelligence_ (e.g. NN training setup, data and algo) is the function that receives the task description and generates _skill_ function in the output. There are different levels of intelligence (effectiveness of skill generation), but this ARC article defines the optimal intelligence function. We can discuss whether LLM training is synthesis of skill function for one task, or is it the ...  
  > **username_0**: ... the synthesis of the set of skill function for the set of tasks and with the choice operators, that allows the recognition of the task and routing the execution to the right function. Or we can hypothesize, that LLM training can be training for intelligence - where LLM can to skill synthesis on the fly. But always - LLM training is the synthesis of the _skill_ or _intelligence_ function (meta-function).  
  > **username_0**: Each synthesis of function wrt the constraints/specification (the correctness of task execution for the _skill_ function and the efficiency of skill generation for the _intelligence_ function) involved two steps: 1) generation or the function; 2) proof that the generated function is correct wrt specification. The steps can be fused, if the proof is being done constructively. We can posit different things about generation of the function, but it is clear, that the proof step is incomputable un undecidable as it is the subcase of FOL SAT (of SAT of Second Order Logic or even more so for HOL).  
  > **username_0**: It is about the proof step alone, but one can conclude from e experience that usually the first step (generation) is even more complex. So - generally yes - LLM training hides function synthesis and hence - LLM training hides the efforts to solve incomputable problem. Of course, many practical applications degenerate to the finite and computable cases and it may be great or even sufficient for most cases of applications. But wall there is. And there are certainly tasks where it can be hitting this wall. Ah, an not to mention the more earthly encounters with NP complete problems.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-26. Score: 2
  
  The so called Turing-barrier is a hard theoretical limit that no system, biological or artificial, can surpass. The so called training-wall observed in the latest large AI systems arises from practical and engineering limitations rather than theoretical barriers, such as diminishing returns, architectural bottlenecks, data exhaustion, energy and cost constraints, and lack of reasoning/abstraction/out-of-distribution universally despite their ability to mimic such behavior in narrow contexts. There's no proof that the way to solve any of these practical limitations is undecidable by humans intrinsically, they're just typical complex social and engineering decision problems facing various kinds of *complexity* for trade-offs beyond mere theoretic *computability*.
  
  While the Turing barrier cannot be broken, the training wall may still be shifted or circumvented, but perhaps not completely. For example, some new smarter training paradigm such as hybrid neuro-symbolic system and meta-learning framework could possibly further break some of the above mentioned training walls in the future. Or some new model beyond Transformer is found to be better suited for logical reasoning along with long-term memory attention, at least in some specialized field.
  
  
  ---
  
  