Answers.Body: <p>Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout.  </p>
  
  <p>If you see the empirical results on Table 5 (of the page 5) of the maxout's <a href="https://arxiv.org/pdf/1302.4389.pdf" rel="nofollow noreferrer">original paper</a>, you find that the misclassification rate is only very, very slightly lower than that of dropouts. (2.47 % instead of 2.78%)</p>
  
  <p>That could explain the relatively lower interest in the work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-18T10:47:15.010Z
Answers.Text: Thanks for your answer. Cutting the error from 2.78% down to 2.47% is actually a 11% decrease in the error rate. Not a game changer but still interesting and definitely not tiny.
Answers.UserDisplayName: user209974
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-18T15:03:19.753Z
Answers.Text: Fair point.  If that is of interest to your specific application, it certainly is worth investigating.
Answers.UserDisplayName: Amrinder Arora
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T20:27:16.387Z
Answers.LastActivityDate: 2019-03-01T20:27:16.387Z
Answers.UserDisplayName: Amrinder Arora
Answers.Body: <p>A few years later, stumbling on my own question, I feel I could share my view on that point.</p>
  <p>I think it boils down eventually to a question of efficiency. There is nothing wrong per se with maxout; it is a distant cousin of ReLU that is in theory more capable. But the crux of the matter is that to produce a single scalar output, maxout consumes <span class="math-container">$k$</span> neurons, <span class="math-container">$k\geq 2$</span>. The minimum price to pay to replace a standard activation function like ReLU with maxout while keeping the latent dimension is doubling the width of the scalar product. It is rather steep.</p>
  <p>Though, a nice property of maxout is that it tends to keep the signal very much alive, instead of killing it like ReLU does. That property might prove to be important enough for maxout to flourish in some niche market despite its inherent wastefulness.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-12T21:11:24.023Z
Answers.LastActivityDate: 2022-08-12T21:11:24.023Z
Answers.UserDisplayName: user209974
Body: <p><a href="https://arxiv.org/abs/1302.4389" rel="noreferrer">Maxout networks</a> were a simple yet brilliant idea of Goodfellow et al. from 2013 to max feature maps to get a universal approximator of convex activations. The design was tailored for use in conjunction with dropout (then recently introduced) and resulted of course in state-of-the-art results on benchmarks like CIFAR-10 and SVHN.</p>
  
  <p>Five years later, dropout is definitely still in the game, but what about maxout? The paper is still widely cited in recent papers according to Google Scholar, but it seems barely any are actually using the technique.</p>
  
  <p>So is maxout a thing of the past, and if so, why — what made it a top performer in 2013 but not in 2018?</p>
  
Tags: <deep-learning><dropout>
Title: 5 years later, are maxout networks dead, and why?
UserDisplayName: user209974
ThreadText: # 5 years later, are maxout networks dead, and why?
  
  (Asked by: username_0 on 2018-07-10)
  
  [Maxout networks](https://arxiv.org/abs/1302.4389) were a simple yet brilliant idea of Goodfellow et al. from 2013 to max feature maps to get a universal approximator of convex activations. The design was tailored for use in conjunction with dropout (then recently introduced) and resulted of course in state-of-the-art results on benchmarks like CIFAR-10 and SVHN.
  
  Five years later, dropout is definitely still in the game, but what about maxout? The paper is still widely cited in recent papers according to Google Scholar, but it seems barely any are actually using the technique.
  
  So is maxout a thing of the past, and if so, why — what made it a top performer in 2013 but not in 2018?
  
  **Tags:** deep-learning, dropout
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-01. Score: 3
  
  Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout.
  
  If you see the empirical results on Table 5 (of the page 5) of the maxout's [original paper](https://arxiv.org/pdf/1302.4389.pdf), you find that the misclassification rate is only very, very slightly lower than that of dropouts. (2.47 % instead of 2.78%)
  
  That could explain the relatively lower interest in the work.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. Cutting the error from 2.78% down to 2.47% is actually a 11% decrease in the error rate. Not a game changer but still interesting and definitely not tiny.  
  > **username_1**: Fair point.  If that is of interest to your specific application, it certainly is worth investigating.  
  
  ---
  
  ### Answer by username_0 on 2022-08-12. Score: 1
  
  A few years later, stumbling on my own question, I feel I could share my view on that point.
  
  I think it boils down eventually to a question of efficiency. There is nothing wrong per se with maxout; it is a distant cousin of ReLU that is in theory more capable. But the crux of the matter is that to produce a single scalar output, maxout consumes $k$ neurons, $k\geq 2$. The minimum price to pay to replace a standard activation function like ReLU with maxout while keeping the latent dimension is doubling the width of the scalar product. It is rather steep.
  
  Though, a nice property of maxout is that it tends to keep the signal very much alive, instead of killing it like ReLU does. That property might prove to be important enough for maxout to flourish in some niche market despite its inherent wastefulness.
  
  
  ---
  
  