Answers.Body: <blockquote>
  <p>My feeling from what I have generally read about machine learning approaches is that these adapt better to more complex problems with higher dimensionalities.</p>
  </blockquote>
  <p>Your intuition seems to align with the empirical results in these domains in the last few years. Expert System are <em>in general</em> not used for domains like Computer Vision, but there may be exceptions do this.</p>
  <blockquote>
  <p>Specifically, I am currently trying to make a general comparison (if possible) of expert systems with machine learning systems across dimensions of efficiency and complexity</p>
  </blockquote>
  <p>I will try to provide you with an approach to systematically compare them.
  You should create a set of tasks <span class="math-container">$T$</span> to evaluate your comparison. I will assume your metric to evaluate on is accuracy, but you can of course replace this with any score function of your choice.</p>
  <p><strong>Space Complexity</strong></p>
  <p>For all tasks implement the expert system and ML model you want to compare. Compare the amount of space they take and average this by the number of tasks to get an estimate of the complexity for each.</p>
  <p><span class="math-container">$ \hat{E}[\mathrm{space}] = \frac{1}{T} \sum_{t\in T} \mathrm{space}(t)$</span></p>
  <p><strong>Time Complexity</strong></p>
  <p>Similar to measuring space, but now time each of your model and expert system. You could do so for different functions on each of your tasks such as <strong>training</strong> or <strong>predicting</strong>. Training is more difficult, because if you're doing supervised learning for example you don't know when you will stop since more training time might be better. So you should set a time or some criteria of convergence and list them under your assumptions.</p>
  <p><span class="math-container">$ \hat{E}[\mathrm{time}] = \frac{1}{T} \sum_{t\in T} \mathrm{time}(\mathrm{predict}(t))$</span></p>
  <p><strong>Run-time-efficiency</strong></p>
  <p>Here is where things get tricky, because there isn't a single unified expression for this that I'm aware of. So I would suggest to evaluate this based on dividing the <strong>accuracy</strong> by the <strong>approximate number of operations</strong>. This could be done multiple times and averaged for example if training a neural net for each epoch.
  To reflect the actual run time in practice more accurately you might also just want to add a time comparison in the form of <strong>accuracy</strong> divided by <strong>runtime on fixed hardware</strong>.</p>
  <p>All of this provides of course no strong theoretical explanation, but is more of a <strong>study to provide some empirical data</strong>. If this is done on a large enough scale your results become a lot more meaningful. Scale here means the number of different model types and expert systems and tasks you're testing on.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-02T23:09:15.913Z
Answers.Text: Thank you @tnfru! I took a while to respond, but your answer gave me good insights
Answers.UserDisplayName: Chris
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-06T18:48:46.467Z
Answers.LastActivityDate: 2021-09-06T18:55:09.073Z
Answers.LastEditDate: 2021-09-06T18:55:09.073Z
Answers.UserDisplayName: tnfru
Body: <p>For part of a paper I am writing on Clinical Decision Support Systems (computer-aided medical decision making, e.g. diagnosis, treatment), I am trying to compare Expert Systems with systems based on Machine Learning approaches (Deep Learning, Artificial Neural Networks, etc.).</p>
  <p>Specifically, I am currently trying to make a general comparison (if possible) of expert systems with machine learning systems across dimensions of efficiency and complexity, i.e.</p>
  <ul>
  <li>run-time-efficiency</li>
  <li>time complexity</li>
  <li>space complexity</li>
  </ul>
  <p>My current line of thinking, after having tried to find literature with limited success, is that, in the case where one is trying to answer questions in a very specific, limited, domain that only requires a few rules (for an expert system), expert systems are relatively &quot;cheap&quot; in terms of these three criteria. However, when a problem/domain becomes more complex, expert systems seem to suffer from the fact that the number of rules needed &quot;explodes&quot;, which, I would think, could lead to things such as large search trees or other problems. My feeling from what I have generally read about machine learning approaches is that these adapt better to more complex problems with higher dimensionalities.</p>
  <p>I would like to find some information that either confirms/backs up my general impression, or guides me to some other understanding of this.</p>
  <p>Unfortunately, I can't seem to find any sources that specifically deal with this kind of comparison. I'm not sure if I this is because my problem statement is to wide/vague, I am not searching correctly, there just isn't much literature, or my question doesn't make sense.</p>
  <p>Some of the sources I <em>did</em> manage to find are:</p>
  <blockquote>
  <p>Expert systems are still used and important in areas such as robotics
  and monitoring. However, the complexity of advanced rules systems can
  lead to performance issues. ANNs are currently managing to overcome
  such performance issues through scale-out.</p>
  <p>Source: <a href="https://www.forbes.com/sites/davidteich/2018/07/06/management-ai-types-of-machine-learning-systems/?sh=17c3413f32fb" rel="nofollow noreferrer">Forbes</a></p>
  </blockquote>
  <p>Unfortunately, this is the most <em>explicit</em> source I've found. However, it doesn't really provide any details on which this claim could backed up, nor would I consider this a solid source, especially not in an academic setting.</p>
  <blockquote>
  <p>Checking for the logical consistency of a set of interrelated logical rules results in the formulation of a
  satisfiability (SAT) problem [Bezem, 1988]. If one assumes only binary variables, say n of them, then the
  corresponding search space is of size 2n
  . That is, it can become very large quickly. This is an NP-complete problem
  very susceptible to the “dimensionality curse” problem [Hansen and Jaumard, 1990]</p>
  <p>Source: <a href="http://fs.unm.edu/neut/ASystematicSurveyOfComputer.pdf" rel="nofollow noreferrer">Yanase J, and Triantaphyllou E, 2019, A Systematic Survey of Computer-Aided
  Diagnosis in Medicine: Past and Present Developments</a>, page 7</p>
  </blockquote>
  <p>This mentions &quot;dimensionality curse&quot;, but in the context of checking for logical consistency of the rules of an expert system, and not really in the context of run-time-efficiency &amp; complexity.</p>
  <p>I have found numerous other articles comparing expert systems and machine learning approaches, e.g. <a href="http://proceedings.mlr.press/v85/ravuri18a/ravuri18a.pdf" rel="nofollow noreferrer">Ravuri et al., 2019, Learning from the experts:
  From expert systems to machine-learned diagnosis models</a>, but none of them, from what I have seen, compare expert systems and machine learning approaches across the dimensions I am interested in.</p>
  <p>Would anyone be able to provide some input on what would be aspects in comparing expert systems and machine learning approaches in terms of the efficiency and complexity criteria listed above, and/or, be able to point me in the right direction?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-06T14:43:18.707Z
Comments.Text: To me, this post is a little too broad, although you seem to understand that when you say "trying to make a general comparison (if possible)". There are many machine learning techniques and there are probably different "expert systems", so I don't think it's possible to provide an accurate answer to this question without writing a long review. I would encourage you to try to narrow the scope of this post/answer. Moreover, it's important to note that "time complexity", "space complexity", "(run-time) efficiency" may refer to different things.
Comments.UserDisplayName: nbro
Tags: <machine-learning><comparison><efficiency><expert-systems><decision-support-system>
Title: A comparison of Expert Systems and Machine Learning approaches in terms of run-time-efficiency and time/space complexity
UserDisplayName: Chris
ThreadText: # A comparison of Expert Systems and Machine Learning approaches in terms of run-time-efficiency and time/space complexity
  
  (Asked by: username_0 on 2021-08-21)
  
  For part of a paper I am writing on Clinical Decision Support Systems (computer-aided medical decision making, e.g. diagnosis, treatment), I am trying to compare Expert Systems with systems based on Machine Learning approaches (Deep Learning, Artificial Neural Networks, etc.).
  
  Specifically, I am currently trying to make a general comparison (if possible) of expert systems with machine learning systems across dimensions of efficiency and complexity, i.e.
  
  * run-time-efficiency
  * time complexity
  * space complexity
  
  My current line of thinking, after having tried to find literature with limited success, is that, in the case where one is trying to answer questions in a very specific, limited, domain that only requires a few rules (for an expert system), expert systems are relatively "cheap" in terms of these three criteria. However, when a problem/domain becomes more complex, expert systems seem to suffer from the fact that the number of rules needed "explodes", which, I would think, could lead to things such as large search trees or other problems. My feeling from what I have generally read about machine learning approaches is that these adapt better to more complex problems with higher dimensionalities.
  
  I would like to find some information that either confirms/backs up my general impression, or guides me to some other understanding of this.
  
  Unfortunately, I can't seem to find any sources that specifically deal with this kind of comparison. I'm not sure if I this is because my problem statement is to wide/vague, I am not searching correctly, there just isn't much literature, or my question doesn't make sense.
  
  Some of the sources I *did* manage to find are:
  
  > Expert systems are still used and important in areas such as robotics
  > and monitoring. However, the complexity of advanced rules systems can
  > lead to performance issues. ANNs are currently managing to overcome
  > such performance issues through scale-out.
  >
  > Source: [Forbes](https://www.forbes.com/sites/davidteich/2018/07/06/management-ai-types-of-machine-learning-systems/?sh=17c3413f32fb)
  
  Unfortunately, this is the most *explicit* source I've found. However, it doesn't really provide any details on which this claim could backed up, nor would I consider this a solid source, especially not in an academic setting.
  
  > Checking for the logical consistency of a set of interrelated logical rules results in the formulation of a
  > satisfiability (SAT) problem [Bezem, 1988]. If one assumes only binary variables, say n of them, then the
  > corresponding search space is of size 2n
  > . That is, it can become very large quickly. This is an NP-complete problem
  > very susceptible to the “dimensionality curse” problem [Hansen and Jaumard, 1990]
  >
  > Source: [Yanase J, and Triantaphyllou E, 2019, A Systematic Survey of Computer-Aided
  > Diagnosis in Medicine: Past and Present Developments](http://fs.unm.edu/neut/ASystematicSurveyOfComputer.pdf), page 7
  
  This mentions "dimensionality curse", but in the context of checking for logical consistency of the rules of an expert system, and not really in the context of run-time-efficiency & complexity.
  
  I have found numerous other articles comparing expert systems and machine learning approaches, e.g. [Ravuri et al., 2019, Learning from the experts:
  From expert systems to machine-learned diagnosis models](http://proceedings.mlr.press/v85/ravuri18a/ravuri18a.pdf), but none of them, from what I have seen, compare expert systems and machine learning approaches across the dimensions I am interested in.
  
  Would anyone be able to provide some input on what would be aspects in comparing expert systems and machine learning approaches in terms of the efficiency and complexity criteria listed above, and/or, be able to point me in the right direction?
  
  **Tags:** machine-learning, comparison, efficiency, expert-systems, decision-support-system
  
  > **Comments:**
  > **username_1**: To me, this post is a little too broad, although you seem to understand that when you say "trying to make a general comparison (if possible)". There are many machine learning techniques and there are probably different "expert systems", so I don't think it's possible to provide an accurate answer to this question without writing a long review. I would encourage you to try to narrow the scope of this post/answer. Moreover, it's important to note that "time complexity", "space complexity", "(run-time) efficiency" may refer to different things.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-09-06. Score: 1 | Accepted answer
  
  > My feeling from what I have generally read about machine learning approaches is that these adapt better to more complex problems with higher dimensionalities.
  
  Your intuition seems to align with the empirical results in these domains in the last few years. Expert System are *in general* not used for domains like Computer Vision, but there may be exceptions do this.
  
  > Specifically, I am currently trying to make a general comparison (if possible) of expert systems with machine learning systems across dimensions of efficiency and complexity
  
  I will try to provide you with an approach to systematically compare them.
  You should create a set of tasks $T$ to evaluate your comparison. I will assume your metric to evaluate on is accuracy, but you can of course replace this with any score function of your choice.
  
  **Space Complexity**
  
  For all tasks implement the expert system and ML model you want to compare. Compare the amount of space they take and average this by the number of tasks to get an estimate of the complexity for each.
  
  $ \hat{E}[\mathrm{space}] = \frac{1}{T} \sum\_{t\in T} \mathrm{space}(t)$
  
  **Time Complexity**
  
  Similar to measuring space, but now time each of your model and expert system. You could do so for different functions on each of your tasks such as **training** or **predicting**. Training is more difficult, because if you're doing supervised learning for example you don't know when you will stop since more training time might be better. So you should set a time or some criteria of convergence and list them under your assumptions.
  
  $ \hat{E}[\mathrm{time}] = \frac{1}{T} \sum\_{t\in T} \mathrm{time}(\mathrm{predict}(t))$
  
  **Run-time-efficiency**
  
  Here is where things get tricky, because there isn't a single unified expression for this that I'm aware of. So I would suggest to evaluate this based on dividing the **accuracy** by the **approximate number of operations**. This could be done multiple times and averaged for example if training a neural net for each epoch.
  To reflect the actual run time in practice more accurately you might also just want to add a time comparison in the form of **accuracy** divided by **runtime on fixed hardware**.
  
  All of this provides of course no strong theoretical explanation, but is more of a **study to provide some empirical data**. If this is done on a large enough scale your results become a lot more meaningful. Scale here means the number of different model types and expert systems and tasks you're testing on.
  
  > **Comments:**
  > **username_0**: Thank you @username_2! I took a while to respond, but your answer gave me good insights  
  
  ---
  
  