Answers.Body: <p>Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.</p>
  
  <p>Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.</p>
  
  <p>Each optimisation problem has a <a href="https://www.youtube.com/watch?v=JPTIZbGGYmk" rel="nofollow noreferrer">landscape</a> or a fitness <a href="https://en.wikipedia.org/wiki/Fitness_landscape" rel="nofollow noreferrer">landscape</a>. Computer science has some nice toy <a href="https://complexitylabs.io/fitness-landscapes/" rel="nofollow noreferrer">landscapes</a>. </p>
  
  <p>There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's <a href="http://www.kmalan.co.za/" rel="nofollow noreferrer">work</a>.</p>
  
  <p>Once you can <a href="https://repository.up.ac.za/bitstream/handle/2263/37155/Malan_Survey_2013.pdf;sequence=1" rel="nofollow noreferrer">identify</a> the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research. </p>
  
  <p>For example some researchers are working on how different evolutionary algorithms handle different <a href="https://link.springer.com/chapter/10.1007/10721187_1" rel="nofollow noreferrer">landscapes</a>, or <a href="https://repository.up.ac.za/handle/2263/66262" rel="nofollow noreferrer">neural networks</a> handle different landscapes.</p>
  
  <p>Start by exploring the types of machine learning problems.
  Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-19T09:09:05.703Z
Answers.LastActivityDate: 2019-01-19T09:09:05.703Z
Answers.UserDisplayName: Jason
Body: <p>I have seen a few articles about neural nets. Mostly they went along these lines: we tried these architectures, these meta parameters, we trained it for <span class="math-container">$x$</span> hours on <span class="math-container">$y$</span> CPUs, and it gave us these results that are 0.1% better than state of the art.</p>
  <p>What I am interested in is whether there exists (at least as a work in progress) a framework that gives some explanation why is some architecture better than other, what makes one activation function more suitable for image recognition than another, etc.</p>
  <p>Do you have some tips about where to start? I would prefer something more systematic than a Google search (a book, a list of key articles is ideal).</p>
  
Tags: <neural-networks><reference-request><hyperparameter-optimization>
Title: How can I systematically learn about the theory of neural networks?
UserDisplayName: Josef Ondrej
ThreadText: # How can I systematically learn about the theory of neural networks?
  
  (Asked by: username_0 on 2019-01-18)
  
  I have seen a few articles about neural nets. Mostly they went along these lines: we tried these architectures, these meta parameters, we trained it for $x$ hours on $y$ CPUs, and it gave us these results that are 0.1% better than state of the art.
  
  What I am interested in is whether there exists (at least as a work in progress) a framework that gives some explanation why is some architecture better than other, what makes one activation function more suitable for image recognition than another, etc.
  
  Do you have some tips about where to start? I would prefer something more systematic than a Google search (a book, a list of key articles is ideal).
  
  **Tags:** neural-networks, reference-request, hyperparameter-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-19. Score: 1 | Accepted answer
  
  Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.
  
  Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.
  
  Each optimisation problem has a [landscape](https://www.youtube.com/watch?v=JPTIZbGGYmk) or a fitness [landscape](https://en.wikipedia.org/wiki/Fitness_landscape). Computer science has some nice toy [landscapes](https://complexitylabs.io/fitness-landscapes/).
  
  There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's [work](http://www.kmalan.co.za/).
  
  Once you can [identify](https://repository.up.ac.za/bitstream/handle/2263/37155/Malan_Survey_2013.pdf;sequence=1) the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research.
  
  For example some researchers are working on how different evolutionary algorithms handle different [landscapes](https://link.springer.com/chapter/10.1007/10721187_1), or [neural networks](https://repository.up.ac.za/handle/2263/66262) handle different landscapes.
  
  Start by exploring the types of machine learning problems.
  Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces.
  
  
  ---
  
  