Answers.Body: <p>Regularisation (at least, <span class="math-container">$L_1$</span> and <span class="math-container">$L_2$</span>) can be viewed as an application of Occam's razor. Regularization is widely used in ML and studied in learning theory (see, for example, the <a href="http://koza.if.uj.edu.pl/%7Ekrzemien/radFarm2021/materials/vapnik-principles-of-risk-minimization-for-learning-theory.pdf" rel="nofollow noreferrer">structural risk</a>). Another application of Occam's razor is in <a href="http://www.hutter1.net/ai/uaibook.htm" rel="nofollow noreferrer">the AIXI agent</a>.</p>
  <p>It seems that models like GPT are going against Occam's razor, but I think they are just brute-force solutions that create only hype - it is obvious to me that with more computation, data and model capacity, a model trained to minimize some objective might perform better, but who cares if the generated text contains fewer errors or &quot;seems to make more sense&quot; if you can't still rely on it to do anything that you really want?</p>
  <p>In general, I agree with you that current architectures are more complicated than what they were in the past - but this doesn't necessarily go against Occam's razor, which states (at least in learning theory) that you should choose the simplest hypothesis that is <em>consistent with the data</em> (see also <a href="https://en.wikipedia.org/wiki/Occam_learning" rel="nofollow noreferrer">Occam learning</a>), so it doesn't say to just choose the simplest hypothesis.</p>
  <p>So, I don't think that <em>all</em> ML community is (necessarily) going against Occam's razor.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-11T15:48:03.900Z
Answers.Text: I'll check the references. Thanks for sharing your point of view!
Answers.UserDisplayName: JFR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-11T14:25:26.407Z
Answers.LastActivityDate: 2022-05-11T14:25:26.407Z
Answers.UserDisplayName: nbro
Body: <p>I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as possible. This follows Occam's Razor principle of simplicity. My concern is if this is also true for the ML or, more specifically, the Deep Learning community.</p>
  <p>I try to briefly illustrate this. When designing DL architectures, I find some of them awfully complicated, so many parameters and layers, hand-crafted loss functions that I wonder if that is really necessary. Of course, some of the problems at hand require a non-trivial solution but sometimes it seems a bit too much. From time to time, you see a paper saying that &quot;we did the same but in a less complicated way&quot;. This is cool of course, but I haven't seen many times.</p>
  <p>The question is therefore: Should machine learning engineers/researchers put more effort in simplifying architectures? If interpretability is important, the more simple the model the better (i.e. everybody understand how a linear or sigmoid regressor works but a graph-biased-random-walk-based-parametric-dolphin-topologic transformer not so sure...)</p>
  <p>P.S. The name of transformer is not real ;)</p>
  
Tags: <deep-learning><explainable-ai><ai-field>
Title: Is the Machine Learning community going against Occam's razor?
UserDisplayName: JFR
ThreadText: # Is the Machine Learning community going against Occam's razor?
  
  (Asked by: username_0 on 2022-05-11)
  
  I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as possible. This follows Occam's Razor principle of simplicity. My concern is if this is also true for the ML or, more specifically, the Deep Learning community.
  
  I try to briefly illustrate this. When designing DL architectures, I find some of them awfully complicated, so many parameters and layers, hand-crafted loss functions that I wonder if that is really necessary. Of course, some of the problems at hand require a non-trivial solution but sometimes it seems a bit too much. From time to time, you see a paper saying that "we did the same but in a less complicated way". This is cool of course, but I haven't seen many times.
  
  The question is therefore: Should machine learning engineers/researchers put more effort in simplifying architectures? If interpretability is important, the more simple the model the better (i.e. everybody understand how a linear or sigmoid regressor works but a graph-biased-random-walk-based-parametric-dolphin-topologic transformer not so sure...)
  
  P.S. The name of transformer is not real ;)
  
  **Tags:** deep-learning, explainable-ai, ai-field
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-11. Score: 1
  
  Regularisation (at least, $L\_1$ and $L\_2$) can be viewed as an application of Occam's razor. Regularization is widely used in ML and studied in learning theory (see, for example, the [structural risk](http://koza.if.uj.edu.pl/%7Ekrzemien/radFarm2021/materials/vapnik-principles-of-risk-minimization-for-learning-theory.pdf)). Another application of Occam's razor is in [the AIXI agent](http://www.hutter1.net/ai/uaibook.htm).
  
  It seems that models like GPT are going against Occam's razor, but I think they are just brute-force solutions that create only hype - it is obvious to me that with more computation, data and model capacity, a model trained to minimize some objective might perform better, but who cares if the generated text contains fewer errors or "seems to make more sense" if you can't still rely on it to do anything that you really want?
  
  In general, I agree with you that current architectures are more complicated than what they were in the past - but this doesn't necessarily go against Occam's razor, which states (at least in learning theory) that you should choose the simplest hypothesis that is *consistent with the data* (see also [Occam learning](https://en.wikipedia.org/wiki/Occam_learning)), so it doesn't say to just choose the simplest hypothesis.
  
  So, I don't think that *all* ML community is (necessarily) going against Occam's razor.
  
  > **Comments:**
  > **username_0**: I'll check the references. Thanks for sharing your point of view!  
  
  ---
  
  