Answers.Body: <p>As Matthew Graves explained in another answer No free lunch theorem confirms the flexibility - efficiency trade-off. However, this theorem is describing a situation where you have a set of completely independent tasks. This often doesn't hold, as many different problems are equivalent in their core or at least have some overlap. Then you can do something called "transfer learning", which means that by training to solve one task you also learn something about solving another one (or possibly multiple different tasks).</p>
  
  <p>For example in <a href="https://arxiv.org/abs/1511.06295" rel="noreferrer">Policy Distillation</a> by Rusu et al. they managed to "distill" knowledge from different expert networks into one general network which in the end outperformed each of the experts. The experts were trained for specific tasks while the generalist learned the final policy from these "teachers".</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-29T13:56:05.997Z
Answers.LastActivityDate: 2016-11-29T13:56:05.997Z
Answers.UserDisplayName: Peter Hroššo
Answers.Body: <p>It would appear so.  One example, albeit not specifically AI related, is seen in the difference between digital computers and <a href="https://en.wikipedia.org/wiki/Analog_computer" rel="nofollow">analog computers</a>.  Pretty much everything we think of as a "computer" today is a digital computer with a von Neumann architecture.  And that's because the things are so general purpose that they can be easily programmed to do, essentially, anything.  But analog computers can (or could, back in the 60's or thereabouts) solve some types of problems faster than a digital computer.  But they fell out of favor exactly due to that lack of flexibility.  Nobody wants to hand-wire circuits with op-amps and comparators to solve for <em>y</em>.  </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-19T15:06:34.227Z
Answers.LastActivityDate: 2016-09-19T15:06:34.227Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>The cleanest result we have on this issue is the <a href="https://en.wikipedia.org/wiki/No_free_lunch_theorem" rel="noreferrer">"no free lunch" theorem</a>. Basically, in order to make a system perform better at a specific task, you have to degrade its performance on other tasks, and so there is a flexibility-efficiency tradeoff.</p>
  
  <p>But to the broader question, or whether or not your thinking is correct, I think it pays to look more closely at what you mean by a "narrow intelligence." The AI systems that we have that play Go and drive cars did <em>not</em> pop into existence able to do those things; they slowly learned how through lots and lots of training examples and a well-chosen architecture that mirrors the problem domain.</p>
  
  <p>That is, "neural networks" as a methodology seems 'general' in a meaningful way; one could imagine that a general intelligence could be formed by solving the meta-learning problem (that is, learning the architecture that best suits a particular problem while learning the weights for that problem from training data).</p>
  
  <p>Even in that case, there will still be a flexibility-efficiency tradeoff; the general intelligence that's allowed to vary its architecture will be able to solve many different problems, but will take some time to discover what problem it's facing. An intelligence locked into a particular architecture will perform well on problems that architecture is well-suited for (better than the general, since it doesn't need to discover) but less well on other problems it isn't as well-suited for.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-22T04:34:42.003Z
Answers.Text: 'No Free Lunch' (NFL) theorems are generally framed in terms of black box systems. By using whitebox descriptions of the problem to be solved and/or metacognition about the solution process, it may be possible to circumvent the NFL. See also my answer to this question https://ai.stackexchange.com/questions/1751/what-are-hyper-heuristics
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-19T14:44:48.200Z
Answers.LastActivityDate: 2016-09-19T14:44:48.200Z
Answers.UserDisplayName: Matthew Gray
Body: <p>A "general intelligence" may be capable of learning a lot of different things, but possessing capability does not equal actually having it. The "AGI" must learn...and that learning process can take time. If you want an AGI to drive a car or play Go, you have to find some way of "teaching" it. Keep in mind that we have never built AGIs, so we don't know how long the training process can be, but it would be safe to assume pessimistic estimates.</p>
  
  <p>Contrast that to a "narrow intelligence". The narrow AI already knows how to drive a car or play Go. It has been programmed to be very excellent at one specific task. You don't need to worry about training the machine, because it has already been pre-trained.</p>
  
  <p>A "general intelligence" seems to be more flexible than a "narrow intelligence". You could buy an AGI and have it drive a car <em>and</em> play Go. And if you are willing to do more training, you can even teach it a new trick: <em>how to bake a cake</em>. I don't have to worry about unexpected tasks coming up, since the AGI will <em>eventually</em> figure out how to do it, given enough training time. I would have to wait a <em>long time</em> though.</p>
  
  <p>A "narrow intelligence" appears to be <em>more efficient</em> at its assigned task, due to it being programmed specifically for that task. It knows exactly what to do, and doesn't have to waste time "learning" (unlike our AGI buddy here). Instead of buying one AGI to handle a bunch of different tasks poorly, I would rather buy a bunch of specialized narrow AIs. Narrow AI #1 drives cars, Narrow AI #2 plays Go, Narrow AI #3 bake cakes, etc. That being said, this is a very brittle approach, since if some unexpected task comes up, none of my narrow AIs would be able to handle it. I'm willing to accept that risk though.</p>
  
  <p>Is my "thinking" correct? Is there a trade-off between flexibility (AGI) and efficiency (narrow AI), like what I have just described above? Or is it theoretically possible for an AGI to be both flexible and efficient?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-04T20:37:13.027Z
Comments.Text: It's not just restricted to "AGI" vs. "strong narrow".  There is a strategy known as "[satisficing](https://en.wikipedia.org/wiki/Satisficing)" in which "good enough" is the best you can achieve b/c objectively optimal decisions cannot be reached.  Where decision time is constrained, on models that can be solved or partially solved, simple heuristics can outperform deep learning.
Comments.UserDisplayName: DukeZhou
Tags: <ai-design><agi><narrow-ai>
Title: Is there a trade-off between flexibility and efficiency?
UserDisplayName: Left SE On 10_6_19
ThreadText: # Is there a trade-off between flexibility and efficiency?
  
  (Asked by: username_0 on 2016-09-18)
  
  A "general intelligence" may be capable of learning a lot of different things, but possessing capability does not equal actually having it. The "AGI" must learn...and that learning process can take time. If you want an AGI to drive a car or play Go, you have to find some way of "teaching" it. Keep in mind that we have never built AGIs, so we don't know how long the training process can be, but it would be safe to assume pessimistic estimates.
  
  Contrast that to a "narrow intelligence". The narrow AI already knows how to drive a car or play Go. It has been programmed to be very excellent at one specific task. You don't need to worry about training the machine, because it has already been pre-trained.
  
  A "general intelligence" seems to be more flexible than a "narrow intelligence". You could buy an AGI and have it drive a car *and* play Go. And if you are willing to do more training, you can even teach it a new trick: *how to bake a cake*. I don't have to worry about unexpected tasks coming up, since the AGI will *eventually* figure out how to do it, given enough training time. I would have to wait a *long time* though.
  
  A "narrow intelligence" appears to be *more efficient* at its assigned task, due to it being programmed specifically for that task. It knows exactly what to do, and doesn't have to waste time "learning" (unlike our AGI buddy here). Instead of buying one AGI to handle a bunch of different tasks poorly, I would rather buy a bunch of specialized narrow AIs. Narrow AI #1 drives cars, Narrow AI #2 plays Go, Narrow AI #3 bake cakes, etc. That being said, this is a very brittle approach, since if some unexpected task comes up, none of my narrow AIs would be able to handle it. I'm willing to accept that risk though.
  
  Is my "thinking" correct? Is there a trade-off between flexibility (AGI) and efficiency (narrow AI), like what I have just described above? Or is it theoretically possible for an AGI to be both flexible and efficient?
  
  **Tags:** ai-design, agi, narrow-ai
  
  > **Comments:**
  > **username_1**: It's not just restricted to "AGI" vs. "strong narrow".  There is a strategy known as "[satisficing](https://en.wikipedia.org/wiki/Satisficing)" in which "good enough" is the best you can achieve b/c objectively optimal decisions cannot be reached.  Where decision time is constrained, on models that can be solved or partially solved, simple heuristics can outperform deep learning.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2016-09-19. Score: 6 | Accepted answer
  
  The cleanest result we have on this issue is the ["no free lunch" theorem](https://en.wikipedia.org/wiki/No_free_lunch_theorem). Basically, in order to make a system perform better at a specific task, you have to degrade its performance on other tasks, and so there is a flexibility-efficiency tradeoff.
  
  But to the broader question, or whether or not your thinking is correct, I think it pays to look more closely at what you mean by a "narrow intelligence." The AI systems that we have that play Go and drive cars did *not* pop into existence able to do those things; they slowly learned how through lots and lots of training examples and a well-chosen architecture that mirrors the problem domain.
  
  That is, "neural networks" as a methodology seems 'general' in a meaningful way; one could imagine that a general intelligence could be formed by solving the meta-learning problem (that is, learning the architecture that best suits a particular problem while learning the weights for that problem from training data).
  
  Even in that case, there will still be a flexibility-efficiency tradeoff; the general intelligence that's allowed to vary its architecture will be able to solve many different problems, but will take some time to discover what problem it's facing. An intelligence locked into a particular architecture will perform well on problems that architecture is well-suited for (better than the general, since it doesn't need to discover) but less well on other problems it isn't as well-suited for.
  
  > **Comments:**
  > **username_5**: 'No Free Lunch' (NFL) theorems are generally framed in terms of black box systems. By using whitebox descriptions of the problem to be solved and/or metacognition about the solution process, it may be possible to circumvent the NFL. See also my answer to this question https://ai.stackexchange.com/questions/1751/what-are-hyper-heuristics  
  
  ---
  
  ### Answer by username_2 on 2016-11-29. Score: 5
  
  As username_4 Graves explained in another answer No free lunch theorem confirms the flexibility - efficiency trade-off. However, this theorem is describing a situation where you have a set of completely independent tasks. This often doesn't hold, as many different problems are equivalent in their core or at least have some overlap. Then you can do something called "transfer learning", which means that by training to solve one task you also learn something about solving another one (or possibly multiple different tasks).
  
  For example in [Policy Distillation](https://arxiv.org/abs/1511.06295) by Rusu et al. they managed to "distill" knowledge from different expert networks into one general network which in the end outperformed each of the experts. The experts were trained for specific tasks while the generalist learned the final policy from these "teachers".
  
  
  ---
  
  ### Answer by username_3 on 2016-09-19. Score: 1
  
  It would appear so. One example, albeit not specifically AI related, is seen in the difference between digital computers and [analog computers](https://en.wikipedia.org/wiki/Analog_computer). Pretty much everything we think of as a "computer" today is a digital computer with a von Neumann architecture. And that's because the things are so general purpose that they can be easily programmed to do, essentially, anything. But analog computers can (or could, back in the 60's or thereabouts) solve some types of problems faster than a digital computer. But they fell out of favor exactly due to that lack of flexibility. Nobody wants to hand-wire circuits with op-amps and comparators to solve for *y*.
  
  
  ---
  
  