Answers.Body: <p>What you are looking for is called Computational Learning Theory. It directly deals with all of these questions, albeit using somewhat different terminology. A good place to get started is here:</p>
  <p><a href="https://en.wikipedia.org/wiki/Computational_learning_theory" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Computational_learning_theory</a>
  <a href="https://en.wikipedia.org/wiki/Probably_approximately_correct_learning" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Probably_approximately_correct_learning</a></p>
  <p>If you are willing to pay, this book is a decent resource: <a href="https://rads.stackoverflow.com/amzn/click/com/0262111934" rel="nofollow noreferrer" rel="nofollow noreferrer">https://www.amazon.com/Introduction-Computational-Learning-Theory-Press/dp/0262111934</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-12T16:02:06.740Z
Answers.LastActivityDate: 2022-12-12T16:02:06.740Z
Answers.UserDisplayName: chessprogrammer
Body: <p>The essential characteristic of machine learning is that an algorithm can discover the behavior of a system on its own. Neural networks are a foremost example of this.</p>
  <p>But what property do neural networks have, for them to be able to discover a <em>range</em> of patterns?</p>
  <p>Has the class of all algorithms with that property been studied?</p>
  <p>I think any function is trainable if it has a parameter of variation, and a rule by which the parameter varies.</p>
  <p>A trivial example is that the equation <code>y = 3</code> is “trainable” by this rule:</p>
  <pre><code>If y &lt; 3, add 1.
  If y &gt; 3, subtract 1.
  </code></pre>
  <p>Starting at <code>y = 0</code>, you would succeed in finding the right value for y.</p>
  <p>I would like to view the concept of “trainable functions” from the perspective of set theory or category theory, but I am not experienced in them yet.</p>
  <p>One core premise I am working with is that all of mathematics can be defined in terms of sets, so any mathematical concept - like “+”, or a function - can be thought of as some set. Therefore, any aspect of a system can vary - the operation, rather than an numerical value - and the rule by which it varies can be of any type, not just numerical adjustment, but just random selection of a different element in the set.</p>
  <p>This is as far as I’ve gotten in my thinking. I think the next point of observation is how some “sets with a rule of variation” are limited. If you vary the <code>m</code> in <code>y = mx</code>, you can not discover a function like <code>y = x^2</code>.</p>
  <p>The point is to eventually understand why neural networks are so general, compared to other optimization functions.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-11T13:38:32.867Z
Comments.Text: Welcome to the site!  I'm reaching out to help you formulate a better question since you are new to this site.  This is the type of question that would normally get downvoted because it is verbose, is asking multiple questions, and is asking subjective questions.  Check out the help section to learn how to ask good questions.  As a general rule of thumb, people don't have time.  They certainly don't have time to read long wordy posts, so to get responses, make your posts short, clear, and to the point.
Comments.UserDisplayName: Snehal Patel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-12T13:10:45.437Z
Comments.Text: I think I made it into a single question, maybe I’ll need a second edit though. It’s basically, if we identify a key property of neural networks, then study the class of all functions with that property, can we understand why neural networks are more capable than techniques like linear regression, what extra property do they have compared to other optimization techniques. Thanks.
Comments.UserDisplayName: Julius Hamilton
Tags: <machine-learning>
Title: What is the most abstract mathematical treatment of machine learning?
UserDisplayName: Julius Hamilton
ThreadText: # What is the most abstract mathematical treatment of machine learning?
  
  (Asked by: username_0 on 2022-12-11)
  
  The essential characteristic of machine learning is that an algorithm can discover the behavior of a system on its own. Neural networks are a foremost example of this.
  But what property do neural networks have, for them to be able to discover a range of patterns?
  Has the class of all algorithms with that property been studied?
  I think any function is trainable if it has a parameter of variation, and a rule by which the parameter varies.
  A trivial example is that the equation y = 3 is “trainable” by this rule:
  If y < 3, add 1.
  If y > 3, subtract 1.
  
  Starting at y = 0, you would succeed in finding the right value for y.
  I would like to view the concept of “trainable functions” from the perspective of set theory or category theory, but I am not experienced in them yet.
  One core premise I am working with is that all of mathematics can be defined in terms of sets, so any mathematical concept - like “+”, or a function - can be thought of as some set. Therefore, any aspect of a system can vary - the operation, rather than an numerical value - and the rule by which it varies can be of any type, not just numerical adjustment, but just random selection of a different element in the set.
  This is as far as I’ve gotten in my thinking. I think the next point of observation is how some “sets with a rule of variation” are limited. If you vary the m in y = mx, you can not discover a function like y = x^2.
  The point is to eventually understand why neural networks are so general, compared to other optimization functions.
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: Welcome to the site!  I'm reaching out to help you formulate a better question since you are new to this site.  This is the type of question that would normally get downvoted because it is verbose, is asking multiple questions, and is asking subjective questions.  Check out the help section to learn how to ask good questions.  As a general rule of thumb, people don't have time.  They certainly don't have time to read long wordy posts, so to get responses, make your posts short, clear, and to the point.  
  > **username_0**: I think I made it into a single question, maybe I’ll need a second edit though. It’s basically, if we identify a key property of neural networks, then study the class of all functions with that property, can we understand why neural networks are more capable than techniques like linear regression, what extra property do they have compared to other optimization techniques. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-12. Score: 1
  
  What you are looking for is called Computational Learning Theory. It directly deals with all of these questions, albeit using somewhat different terminology. A good place to get started is here:
  
  <https://en.wikipedia.org/wiki/Computational_learning_theory>
  <https://en.wikipedia.org/wiki/Probably_approximately_correct_learning>
  
  If you are willing to pay, this book is a decent resource: [https://www.amazon.com/Introduction-Computational-Learning-Theory-Press/dp/0262111934](https://rads.stackoverflow.com/amzn/click/com/0262111934)
  
  
  ---
  
  