Answers.Body: <p><a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf" rel="nofollow noreferrer">Pattern Recognition And Machine Learning</a> is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today.</p>
  <p>So, in addition, I'd suggest you look at <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> by Ian Goodfellow et al.</p>
  <p>Your concerns about not studying topology, measure theory and group theory are groundless. These sections of math aren't prerequisites in any way, they aren't even discussed anywhere I know.</p>
  <p>Actually, ML theory is more like probability theory and statistics. Especially, statistical learning theory (which is nothing more than probability theory and statistics). I haven't read any books on SLT so have a look at <a href="https://ai.stackexchange.com/a/20358/2444">this answer</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-04T21:55:09.743Z
Answers.Text: "Deep Learning" is not a good book.
Answers.UserDisplayName: Kevin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-01T13:32:18.673Z
Answers.Text: @Kevin, why isn't it a good book? I haven't read it yet but I have seen it recommended many times so I am curious.
Answers.UserDisplayName: heretoinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-10T21:07:50.897Z
Answers.LastActivityDate: 2021-01-16T19:34:15.397Z
Answers.LastEditDate: 2021-01-16T19:34:15.397Z
Answers.UserDisplayName: Michał Słodki
Answers.Body: <p>For a more updated list I'd suggest these:</p>
  <ol>
  <li><a href="https://www.bishopbook.com" rel="nofollow noreferrer">Deep Learning: Foundations and Concepts</a> (2024) by Bishop (same author for the classic)</li>
  <li><a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> by Rich Sutton</li>
  <li><a href="https://www.marl-book.com" rel="nofollow noreferrer">Multi-Agent Reinforcement Learning: Foundations and Modern Approaches</a> (2024)</li>
  <li><a href="https://deeplearningtheory.com" rel="nofollow noreferrer">The Principles of Deep Learning Theory</a></li>
  <li><a href="https://mitpress.mit.edu/9780262048972/foundations-of-computer-vision/" rel="nofollow noreferrer">Foundations of Computer Vision</a> (2024)</li>
  </ol>
  <p>There's also a <a href="https://www.lesswrong.com/posts/t4ZBjAjXk2NqqAqJ7/the-27-papers" rel="nofollow noreferrer">list of 27 papers</a> (reportedly shared by Ilya Sutskever with John Carmack).</p>
  <blockquote>
  <p><em>&quot;If you truly master these, you’ll understand 90% of what matters in modern AI.&quot;</em></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-07T19:11:35.607Z
Answers.LastActivityDate: 2024-10-07T19:11:35.607Z
Answers.UserDisplayName: ofou
Answers.Body: <p>Some of the books that you mention are often used as reference books in introductory courses to machine learning or artificial intelligence.</p>
  <p>For example, if I remember correctly, in my introductory course to machine learning, the professor suggested the book <a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf" rel="noreferrer">Pattern Recognition And Machine Learning</a> (2006) by Bishop, although we never used it during the lessons. This is a good book, but, in my opinion, it covers many topics, such as variational inference or sampling methods, that are not suited for an introductory course.</p>
  <p>The book <a href="http://aima.cs.berkeley.edu/" rel="noreferrer">Artificial Intelligence. A Modern Approach</a>, by Norvig and Russell, definitely does not focus on machine learning, but it covers many other aspects of <strong>artificial intelligence</strong>, such as search, planning, knowledge representation, machine learning, robotics, natural language processing or computer vision. This is probably the book that you should read and use if you want to have an extensive overview of the AI field. Although I never fully read it, I often used it as a reference, as I use the other mentioned book. For instance, during my bachelor's and, more specifically, an introductory course to artificial intelligence, we had used this book as the reference book, but note that there are <a href="https://ai.stackexchange.com/a/25799/2444">other books that provide an extensive overview of the AI field</a>.</p>
  <p>The other two books are not as famous as these two, but they are probably also good books, although their focus may be different.</p>
  <p>There are at least three other books that I think you should also be aware of, given that they also cover the actual <em>theory of learning</em>, aka <strong>(computational) learning theory</strong>, before diving into more specific topics, such as kernel methods.</p>
  <ul>
  <li><a href="http://profsite.um.ac.ir/%7Emonsefi/machine-learning/pdf/Machine-Learning-Tom-Mitchell.pdf" rel="noreferrer">Machine Learning</a> (1997) by Tom Mitchell</li>
  <li><a href="https://mitpress.ublish.com/ereader/7093/?preview=#page/Cover" rel="noreferrer">Foundations of Machine Learning</a> (2nd edition, 2018) by Mehryar Mohri et al.</li>
  <li><a href="https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf" rel="noreferrer">Understanding Machine Learning: From Theory to Algorithms</a> (2014) by Shai Shalev-Shwartz et al.</li>
  </ul>
  <p>You can find more books on learning theory <a href="https://ai.stackexchange.com/a/20358/2444">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-11T10:21:58.013Z
Answers.LastActivityDate: 2021-01-24T21:12:37.150Z
Answers.LastEditDate: 2021-01-24T21:12:37.150Z
Answers.UserDisplayName: nbro
Body: <p>I am looking for a book about machine learning that would suit my physics background. I am more or less familiar with classical and complex analysis, theory of probability, сcalculus of variations, matrix algebra, etc. However, I have not studied topology, measure theory, group theory, and other more advanced topics. I try to find a book that is written neither for beginners, nor for mathematicians.</p>
  <p>Recently, I have read the great book &quot;Statistical inference&quot; written by Casella and Berger. They write in the introduction that &quot;The purpose of this book is to build theoretical statistics (as different from mathematical statistics) from the first principles of probability theory&quot;. So, <em>I am looking for some &quot;theoretical books&quot; about machine learning</em>.</p>
  <p>There are many online courses and brilliant books out there that focus on the practical side of applying machine learning models and using the appropriate libraries. It seems to me that there are no problems with them, but I would like to find a book on theory.</p>
  <p>By now I have skimmed through the following books</p>
  <ul>
  <li><p><a href="https://rads.stackoverflow.com/amzn/click/com/0387310738" rel="noreferrer" rel="nofollow noreferrer">Pattern Recognition And Machine Learning</a></p>
  <p>It looks very nice. The only point of concern is that the book was published in 2006. So, I am not sure about the relevance of the chapters considering neural nets, since this field is developing rather fast.</p>
  </li>
  <li><p><a href="https://rads.stackoverflow.com/amzn/click/com/0387848576" rel="noreferrer" rel="nofollow noreferrer">The elements of statistical learning</a></p>
  <p>This book also seems very good. It covers most of the topics as well as the first book. However, I am feeling that its style is different and I do not know which book will suit me better.</p>
  </li>
  <li><p><a href="https://rads.stackoverflow.com/amzn/click/com/0136042597" rel="noreferrer" rel="nofollow noreferrer">Artificial Intelligence. A Modern Approach</a></p>
  <p>This one covers more recent topics, such as natural language processing. As far as I understand, it represents the view of a computer scientist on machine learning.</p>
  </li>
  <li><p><a href="https://rads.stackoverflow.com/amzn/click/com/0262018020" rel="noreferrer" rel="nofollow noreferrer">Machine Learning A Probabilistic Perspective</a></p>
  <p>Maybe it has a slight bias towards probability theory, which is stated in the title. However, the book looks fascinating as well.</p>
  </li>
  </ul>
  <p>I think that the first or the second book should suit me, but I do not know what decision to make.</p>
  <p>I am sure that I have overlooked some books.</p>
  <p><strong>Are there some other ML books that focus on theory?</strong></p>
  
Tags: <machine-learning><reference-request><computational-learning-theory><books>
Title: What are other examples of theoretical machine learning books?
UserDisplayName: Ilya
ThreadText: # What are other examples of theoretical machine learning books?
  
  (Asked by: username_0 on 2020-09-10)
  
  I am looking for a book about machine learning that would suit my physics background. I am more or less familiar with classical and complex analysis, theory of probability, сcalculus of variations, matrix algebra, etc. However, I have not studied topology, measure theory, group theory, and other more advanced topics. I try to find a book that is written neither for beginners, nor for mathematicians.
  
  Recently, I have read the great book "Statistical inference" written by Casella and Berger. They write in the introduction that "The purpose of this book is to build theoretical statistics (as different from mathematical statistics) from the first principles of probability theory". So, *I am looking for some "theoretical books" about machine learning*.
  
  There are many online courses and brilliant books out there that focus on the practical side of applying machine learning models and using the appropriate libraries. It seems to me that there are no problems with them, but I would like to find a book on theory.
  
  By now I have skimmed through the following books
  
  * [Pattern Recognition And Machine Learning](https://rads.stackoverflow.com/amzn/click/com/0387310738)
  
    It looks very nice. The only point of concern is that the book was published in 2006. So, I am not sure about the relevance of the chapters considering neural nets, since this field is developing rather fast.
  * [The elements of statistical learning](https://rads.stackoverflow.com/amzn/click/com/0387848576)
  
    This book also seems very good. It covers most of the topics as well as the first book. However, I am feeling that its style is different and I do not know which book will suit me better.
  * [Artificial Intelligence. A Modern Approach](https://rads.stackoverflow.com/amzn/click/com/0136042597)
  
    This one covers more recent topics, such as natural language processing. As far as I understand, it represents the view of a computer scientist on machine learning.
  * [Machine Learning A Probabilistic Perspective](https://rads.stackoverflow.com/amzn/click/com/0262018020)
  
    Maybe it has a slight bias towards probability theory, which is stated in the title. However, the book looks fascinating as well.
  
  I think that the first or the second book should suit me, but I do not know what decision to make.
  
  I am sure that I have overlooked some books.
  
  **Are there some other ML books that focus on theory?**
  
  **Tags:** machine-learning, reference-request, computational-learning-theory, books
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2020-09-11. Score: 7 | Accepted answer
  
  Some of the books that you mention are often used as reference books in introductory courses to machine learning or artificial intelligence.
  
  For example, if I remember correctly, in my introductory course to machine learning, the professor suggested the book [Pattern Recognition And Machine Learning](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) (2006) by Bishop, although we never used it during the lessons. This is a good book, but, in my opinion, it covers many topics, such as variational inference or sampling methods, that are not suited for an introductory course.
  
  The book [Artificial Intelligence. A Modern Approach](http://aima.cs.berkeley.edu/), by Norvig and Russell, definitely does not focus on machine learning, but it covers many other aspects of **artificial intelligence**, such as search, planning, knowledge representation, machine learning, robotics, natural language processing or computer vision. This is probably the book that you should read and use if you want to have an extensive overview of the AI field. Although I never fully read it, I often used it as a reference, as I use the other mentioned book. For instance, during my bachelor's and, more specifically, an introductory course to artificial intelligence, we had used this book as the reference book, but note that there are [other books that provide an extensive overview of the AI field](https://ai.stackexchange.com/a/25799/2444).
  
  The other two books are not as famous as these two, but they are probably also good books, although their focus may be different.
  
  There are at least three other books that I think you should also be aware of, given that they also cover the actual *theory of learning*, aka **(computational) learning theory**, before diving into more specific topics, such as kernel methods.
  
  * [Machine Learning](http://profsite.um.ac.ir/%7Emonsefi/machine-learning/pdf/Machine-Learning-Tom-Mitchell.pdf) (1997) by Tom Mitchell
  * [Foundations of Machine Learning](https://mitpress.ublish.com/ereader/7093/?preview=#page/Cover) (2nd edition, 2018) by Mehryar Mohri et al.
  * [Understanding Machine Learning: From Theory to Algorithms](https://www.cs.huji.ac.il/%7Eshais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf) (2014) by Shai Shalev-Shwartz et al.
  
  You can find more books on learning theory [here](https://ai.stackexchange.com/a/20358/2444).
  
  
  ---
  
  ### Answer by username_1 on 2020-09-10. Score: 3
  
  [Pattern Recognition And Machine Learning](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) is a great theoretical book. I don't know anything better on standard ML. I read several pages from it myself and all my colleagues researchers suggest to look there if you are not sure about some concepts. The 2 problems with it are that it's huge and it doesn't cover almost all deep learning models known for today.
  
  So, in addition, I'd suggest you look at [Deep Learning](https://www.deeplearningbook.org/) by Ian Goodfellow et al.
  
  Your concerns about not studying topology, measure theory and group theory are groundless. These sections of math aren't prerequisites in any way, they aren't even discussed anywhere I know.
  
  Actually, ML theory is more like probability theory and statistics. Especially, statistical learning theory (which is nothing more than probability theory and statistics). I haven't read any books on SLT so have a look at [this answer](https://ai.stackexchange.com/a/20358/2444).
  
  > **Comments:**
  > **username_2**: "Deep Learning" is not a good book.  
  > **username_3**: @username_2, why isn't it a good book? I haven't read it yet but I have seen it recommended many times so I am curious.  
  
  ---
  
  ### Answer by username_4 on 2024-10-07. Score: 1
  
  For a more updated list I'd suggest these:
  
  1. [Deep Learning: Foundations and Concepts](https://www.bishopbook.com) (2024) by Bishop (same author for the classic)
  2. [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book-2nd.html) by Rich Sutton
  3. [Multi-Agent Reinforcement Learning: Foundations and Modern Approaches](https://www.marl-book.com) (2024)
  4. [The Principles of Deep Learning Theory](https://deeplearningtheory.com)
  5. [Foundations of Computer Vision](https://mitpress.mit.edu/9780262048972/foundations-of-computer-vision/) (2024)
  
  There's also a [list of 27 papers](https://www.lesswrong.com/posts/t4ZBjAjXk2NqqAqJ7/the-27-papers) (reportedly shared by username_0 Sutskever with John Carmack).
  
  > *"If you truly master these, you’ll understand 90% of what matters in modern AI."*
  
  
  ---
  
  