Answers.Body: <p>Whether Minsky knew or not, it was definitely known to Rosenblatt, as he published those results in his really pioneering report - <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf" rel="nofollow noreferrer">Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms</a>, published in 1961.</p>
  <p>A large majority of academic and industry experts are simply unaware of the &quot;depth&quot; of Rosenblatt's publication on perceptrons, where he not only proved that 3-layer perceptrons (which he called elementary) are universal (check <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=116" rel="nofollow noreferrer">theorem 1 in section 5.2</a>), but where he also provided results on convergence (check <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=127" rel="nofollow noreferrer">theorem 4 in section 5.5</a>) and a statistical mechanics analysis of their generalization capabilities (check <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=144" rel="nofollow noreferrer">chapter 6</a> for the foundational theory, and <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=178" rel="nofollow noreferrer">figure 13 and 14</a> for an application of it, following the analysis in <a href="https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=170" rel="nofollow noreferrer">section 7.1.2</a>).</p>
  <p>It is simply unfortunate that Rosenblatt accidentally died soon after Minsky and Papert's not-so-pioneering 1969 book was published. I believe its misleading influence has regressed AI research for several decades.</p>
  <p>If only Rosenblatt lived longer...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-07T14:59:08.140Z
Answers.Text: Hello. Welcome to AI SE. It may be a good idea to provide the name of the 1962 book by Rosenblatt that you're talking about. It would be very nice if you also specify the sections where he proves "that 3-layer perceptrons (which he called elementary) are universal, but where he also provided results on convergence and a statistical mechanics analysis of their generalization capabilities", as you claim.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-10T22:33:29.687Z
Answers.Text: Thank you, I have edited my post with extra information. I wrongly referred to it as the 1962 book, but it is actually a report published on March 15, 1961. I do not know in which year this was unclassified.
Answers.UserDisplayName: Celestine Lawrence
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-11T08:27:55.780Z
Answers.Text: Thanks for your last edit. It may also be a good idea to remove your last paragraph, as it is controversial and not very related to the specific question on the original post. Please, make also sure that the links to the specific section of the report that I added are correct.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-07T14:42:12.987Z
Answers.LastActivityDate: 2024-07-25T22:00:00.337Z
Answers.LastEditDate: 2024-07-25T22:00:00.337Z
Answers.UserDisplayName: Celestine Lawrence
Answers.Body: <p>There does not appear to be a historical consensus on this.</p>
  <p>The <a href="https://en.wikipedia.org/wiki/Perceptrons_(book)" rel="noreferrer">Wikipedia page on the Perceptrons book</a> (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts).</p>
  <p>However, <a href="http://harveycohen.net/image/perceptron.html" rel="noreferrer">this page</a> gives an account by someone present at the MIT AI lab in 1974, claiming that this was not common knowledge there, alluding to documentation in &quot;Artificial Intelligence Progress Report: Research at the Laboratory in Vision, Language, and other problems of Intelligence&quot; (p31-32) which is claimed to support this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-08-07T09:42:52.590Z
Answers.LastActivityDate: 2021-01-19T00:22:23.393Z
Answers.LastEditDate: 2021-01-19T00:22:23.393Z
Answers.UserDisplayName: NietzscheanAI
Answers.Body: <p>In section <strong>13.2 Other Multilayer Machines</strong> (pp. 231-232) of the book <a href="https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=246" rel="noreferrer">Perceptrons: An Introduction to Computational Geometry</a> (expanded edition, third printing, 1988) Minsky and Papert actually talk about their knowledge of or opinions about the capabilities of what they call the <em>multilayered machines</em> (i.e. perceptrons with many layers or MLPs).</p>
  <blockquote>
  <p><em>Have you considered &quot;perceptrons&quot; with many layers?</em></p>
  <p>Well, we have considered <strong>Gamba machines</strong>, which could be described as &quot;<strong>two layers of perceptron</strong>&quot;. We have not found (by thinking or by studying the literature) any other really interesting class of multilayered machine, at least none whose principles seem to have a signiﬁcant relation to those of the perceptron. To see the force of this qualiﬁcation it is worth pondering the fact, trivial in itself, that <strong>a universal computer could be built entirely out of linear threshold modules</strong>. This does not in any sense reduce the theory of computation and programming to the theory of perceptrons. Some philosophers might like to express the relevant general principle by saying that the computer is so much more than the sum of its parts that the computer scientist can aﬁord to ignore the nature of the components and consider only their connectivity. More concretely, we would call the student's attention to the following considerations:</p>
  <ol>
  <li><p>Multilayer machines with loops clearly open all the questions of the general theory of automata.</p>
  </li>
  <li><p>A system with no loops but with an order restriction at each layer can compute only predicates of ﬁnite order.</p>
  </li>
  <li><p>On the other hand, if there is no restriction except for the absence of loops, the monster of vacuous generality once more raises its head.</p>
  </li>
  </ol>
  <p>The problem of extension is not merely technical. It is also strategic. The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations, It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. <strong>There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgment that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting
  &quot;learning theorem&quot; for the multilayered machine will be found.</strong></p>
  </blockquote>
  <p>So, let me address your first question directly.</p>
  <blockquote>
  <p>Backprop wasn't known at the time, but did they know about manually building multilayer perceptrons?</p>
  </blockquote>
  <p>Yes. They say that Gamba machines could be described as a 2-layer perceptron. For reproducibility, here's the definition of the Gamba machine (section <a href="https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=243" rel="noreferrer"><strong>13.1 Gamba Perceptrons and other Multilayer Linear Machines</strong></a>)</p>
  <p><span class="math-container">\begin{align}
  \psi
  &amp;=
  \left[\sum_{i} \alpha_{i}\left[\sum_{j} \beta_{i j} x_{j}&gt;\theta_{i}\right]&gt;\theta\right] \\
  &amp;=
  \left[\sum_{i} \alpha_{i} \varphi_{i} &gt;\theta \right]
  \end{align}</span>
  See also sections <a href="https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=220" rel="noreferrer"><strong>12.4.4. Layer-Machines</strong></a>.</p>
  <p>So, let's now address your second question.</p>
  <blockquote>
  <p>Did Minsky &amp; Papert know that multilayer perceptrons could solve XOR at the time they wrote the book, albeit not knowing how to train it?</p>
  </blockquote>
  <p>So, according to the first excerpt, their intuition was the <em>virtues</em> of perceptrons would not carry over to MLPs, but they acknowledge that more research was needed to reject or support this hypothesis.</p>
  <p>However, in section <a href="https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=242" rel="noreferrer"><strong>13.0 Introduction</strong></a> of the same book, they write</p>
  <blockquote>
  <p>We believe (but cannot prove) that the <strong>deeper limitations extend also to the variant of the perceptron proposed by A. Gamba</strong>.</p>
  </blockquote>
  <p>So, they believed that the Gamba machine would not have been able to solve the XOR problem.</p>
  <p>However, in the first excerpt, they say that a Turing machine could be built entirely out of linear threshold modules, which seems to be inconsistent with the second excerpt, but that's not really the case because they are not saying how to build a Turing machine out of the linear threshold modules but that just the specific Gamba machine would have the same limitations of the perceptron.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T16:28:45.867Z
Answers.LastActivityDate: 2021-01-19T00:47:41.100Z
Answers.LastEditDate: 2021-01-19T00:47:41.100Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Cold war - maybe he knew?</p>
  <p><a href="https://gwern.net/doc/ai/1966-ivakhnenko.pdf" rel="nofollow noreferrer">https://gwern.net/doc/ai/1966-ivakhnenko.pdf</a></p>
  <p>Cybernetics and forecasting techniques
  by Ivakhnenko, Alekseĭ Grigorʹevich
  <a href="https://archive.org/details/cyberneticsforec0000ivak" rel="nofollow noreferrer">https://archive.org/details/cyberneticsforec0000ivak</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T00:31:36.757Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T13:03:48.640Z
Answers.LastActivityDate: 2023-08-31T13:03:48.640Z
Answers.UserDisplayName: Gareth Thomas
Body: <p>In their famous book entitled <a href="https://rads.stackoverflow.com/amzn/click/com/0262631113" rel="noreferrer" rel="nofollow noreferrer">Perceptrons: An Introduction to Computational Geometry</a>, Minsky and Papert show that a perceptron can't solve the XOR problem. This contributed to the first AI winter, resulting in funding cuts for neural networks. However, now we know that a multilayer perceptron can solve the XOR problem easily.</p>
  
  <p>Backprop wasn't known at the time, but did they know about manually building multilayer perceptrons? Did Minsky &amp; Papert know that multilayer perceptrons could solve XOR at the time they wrote the book, albeit not knowing how to train it?</p>
  
Tags: <multilayer-perceptrons><history><perceptron><ai-winter><xor-problem>
Title: Did Minsky and Papert know that multi-layer perceptrons could solve XOR?
UserDisplayName: rcpinto
ThreadText: # Did Minsky and Papert know that multi-layer perceptrons could solve XOR?
  
  (Asked by: username_0 on 2016-08-04)
  
  In their famous book entitled [Perceptrons: An Introduction to Computational Geometry](https://rads.stackoverflow.com/amzn/click/com/0262631113), Minsky and Papert show that a perceptron can't solve the XOR problem. This contributed to the first AI winter, resulting in funding cuts for neural networks. However, now we know that a multilayer perceptron can solve the XOR problem easily.
  
  Backprop wasn't known at the time, but did they know about manually building multilayer perceptrons? Did Minsky & Papert know that multilayer perceptrons could solve XOR at the time they wrote the book, albeit not knowing how to train it?
  
  **Tags:** multilayer-perceptrons, history, perceptron, ai-winter, xor-problem
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2021-10-07. Score: 14
  
  Whether Minsky knew or not, it was definitely known to Rosenblatt, as he published those results in his really pioneering report - [Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf), published in 1961.
  
  A large majority of academic and industry experts are simply unaware of the "depth" of Rosenblatt's publication on perceptrons, where he not only proved that 3-layer perceptrons (which he called elementary) are universal (check [theorem 1 in section 5.2](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=116)), but where he also provided results on convergence (check [theorem 4 in section 5.5](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=127)) and a statistical mechanics analysis of their generalization capabilities (check [chapter 6](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=144) for the foundational theory, and [figure 13 and 14](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=178) for an application of it, following the analysis in [section 7.1.2](https://apps.dtic.mil/sti/pdfs/AD0256582.pdf#page=170)).
  
  It is simply unfortunate that Rosenblatt accidentally died soon after Minsky and Papert's not-so-pioneering 1969 book was published. I believe its misleading influence has regressed AI research for several decades.
  
  If only Rosenblatt lived longer...
  
  > **Comments:**
  > **username_2**: Hello. Welcome to AI SE. It may be a good idea to provide the name of the 1962 book by Rosenblatt that you're talking about. It would be very nice if you also specify the sections where he proves "that 3-layer perceptrons (which he called elementary) are universal, but where he also provided results on convergence and a statistical mechanics analysis of their generalization capabilities", as you claim.  
  > **username_1**: Thank you, I have edited my post with extra information. I wrongly referred to it as the 1962 book, but it is actually a report published on March 15, 1961. I do not know in which year this was unclassified.  
  > **username_2**: Thanks for your last edit. It may also be a good idea to remove your last paragraph, as it is controversial and not very related to the specific question on the original post. Please, make also sure that the links to the specific section of the report that I added are correct.  
  
  ---
  
  ### Answer by username_3 on 2016-08-07. Score: 13
  
  There does not appear to be a historical consensus on this.
  
  The [Wikipedia page on the Perceptrons book](https://en.wikipedia.org/wiki/Perceptrons_(book)) (which does not come down on either side) gives an argument that the ability of MLPs to compute any Boolean function was widely known at the time (at the very least to McCulloch and Pitts).
  
  However, [this page](http://harveycohen.net/image/perceptron.html) gives an account by someone present at the MIT AI lab in 1974, claiming that this was not common knowledge there, alluding to documentation in "Artificial Intelligence Progress Report: Research at the Laboratory in Vision, Language, and other problems of Intelligence" (p31-32) which is claimed to support this.
  
  
  ---
  
  ### Answer by username_2 on 2020-03-10. Score: 7
  
  In section **13.2 Other Multilayer Machines** (pp. 231-232) of the book [Perceptrons: An Introduction to Computational Geometry](https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=246) (expanded edition, third printing, 1988) Minsky and Papert actually talk about their knowledge of or opinions about the capabilities of what they call the *multilayered machines* (i.e. perceptrons with many layers or MLPs).
  
  > *Have you considered "perceptrons" with many layers?*
  >
  > Well, we have considered **Gamba machines**, which could be described as "**two layers of perceptron**". We have not found (by thinking or by studying the literature) any other really interesting class of multilayered machine, at least none whose principles seem to have a signiﬁcant relation to those of the perceptron. To see the force of this qualiﬁcation it is worth pondering the fact, trivial in itself, that **a universal computer could be built entirely out of linear threshold modules**. This does not in any sense reduce the theory of computation and programming to the theory of perceptrons. Some philosophers might like to express the relevant general principle by saying that the computer is so much more than the sum of its parts that the computer scientist can aﬁord to ignore the nature of the components and consider only their connectivity. More concretely, we would call the student's attention to the following considerations:
  >
  > 1. Multilayer machines with loops clearly open all the questions of the general theory of automata.
  > 2. A system with no loops but with an order restriction at each layer can compute only predicates of ﬁnite order.
  > 3. On the other hand, if there is no restriction except for the absence of loops, the monster of vacuous generality once more raises its head.
  >
  > The problem of extension is not merely technical. It is also strategic. The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations, It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. **There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgment that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting
  > "learning theorem" for the multilayered machine will be found.**
  
  So, let me address your first question directly.
  
  > Backprop wasn't known at the time, but did they know about manually building multilayer perceptrons?
  
  Yes. They say that Gamba machines could be described as a 2-layer perceptron. For reproducibility, here's the definition of the Gamba machine (section [**13.1 Gamba Perceptrons and other Multilayer Linear Machines**](https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=243))
  
  \begin{align}
  \psi
  &=
  \left[\sum\_{i} \alpha\_{i}\left[\sum\_{j} \beta\_{i j} x\_{j}>\theta\_{i}\right]>\theta\right] \\
  &=
  \left[\sum\_{i} \alpha\_{i} \varphi\_{i} >\theta \right]
  \end{align}
  See also sections [**12.4.4. Layer-Machines**](https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=220).
  
  So, let's now address your second question.
  
  > Did Minsky & Papert know that multilayer perceptrons could solve XOR at the time they wrote the book, albeit not knowing how to train it?
  
  So, according to the first excerpt, their intuition was the *virtues* of perceptrons would not carry over to MLPs, but they acknowledge that more research was needed to reject or support this hypothesis.
  
  However, in section [**13.0 Introduction**](https://russell-davidson.arts.mcgill.ca/e706/Perceptrons.pdf#page=242) of the same book, they write
  
  > We believe (but cannot prove) that the **deeper limitations extend also to the variant of the perceptron proposed by A. Gamba**.
  
  So, they believed that the Gamba machine would not have been able to solve the XOR problem.
  
  However, in the first excerpt, they say that a Turing machine could be built entirely out of linear threshold modules, which seems to be inconsistent with the second excerpt, but that's not really the case because they are not saying how to build a Turing machine out of the linear threshold modules but that just the specific Gamba machine would have the same limitations of the perceptron.
  
  
  ---
  
  ### Answer by username_4 on 2023-08-31. Score: 0
  
  Cold war - maybe he knew?
  
  <https://gwern.net/doc/ai/1966-ivakhnenko.pdf>
  
  Cybernetics and forecasting techniques
  by Ivakhnenko, Alekseĭ Grigorʹevich
  <https://archive.org/details/cyberneticsforec0000ivak>
  
  > **Comments:**
  > **username_5**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  