Answers.Body: <p>Konstantine, I assume you refer to plain MLP and CNN, without any modifications.</p>
  
  <p>I believe what you ask is how to set both of them up, in order to have the fairest comparison possible.</p>
  
  <p>The way I would do it, is to use their <strong>plain implementations</strong> but <strong>both tuned as much as possible, in every hyperparameter</strong>. Both should work as black boxes that accept the same inputs and give the same outputs.</p>
  
  <p>This will give you insight on the true raw performance of both algorithms.</p>
  
  <p>Hope it helps :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-14T11:24:52.413Z
Answers.Text: I'm not sure what plain implies. My models are sequential models with two hidden layers (for the convolutional network the first hidden layer is composed by two convolutional layers and a pooling layer). The idea of comparing the best out of the two is appealing. Although i don't have a lot of hyperparameters to play with, so I was thinking more of a size comparisons between the two (size in number of neurons, weights, parameters to learn, etc)
Answers.UserDisplayName: Constantine Pat.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-14T11:47:53.980Z
Answers.Text: By "plain" I refer to the most "basic" implementation of both algorithms. I don't see direct evaluation from just comparing the e.g. weights of both. This is because the architecture is different so you are comparing "apples and oranges". What would make sense is to compare the overall performance, not the inner characteristics. This means, don't focus on comparing hyperparameters, just make sure you fine-tune both networks. You should compare their results for the same inputs. Results like accuracy and training/inference time. Hope it makes sense. Kalh tyxh :)
Answers.UserDisplayName: pcko1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-13T18:03:30.690Z
Answers.LastActivityDate: 2018-06-13T18:03:30.690Z
Answers.UserDisplayName: pcko1
Answers.Body: <p>The best best way to monitor an architectures performance would be comparing the resource utilization, model accuracy, value loss and confusion matrix.
  <br>
  e.g VGG16 consumes less system resources in comparison to inception V3
  <br>
  <br>
  There is an <a href="https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/" rel="nofollow noreferrer">article</a> that goes in depth.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-14T11:20:35.057Z
Answers.Text: The article talks more about various metrics of performance, which is not really my concern. The idea of resource utilization is interesting. Any ideas how to compare these in my models? (I use the keras library). Perhaps check the number of parameters to train for each network? (keras outputs the total number of trainable parameters)
Answers.UserDisplayName: Constantine Pat.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-12T08:16:11.630Z
Answers.LastActivityDate: 2018-06-12T08:16:11.630Z
Answers.UserDisplayName: Simbarashe Timothy Motsi
Body: <p>I'm working with deep learning on some EEG data for classification, and I was wondering if there's any systematic/mathematical way to define the architecture of the networks, in order to compare their performance fairly.</p>
  
  <p>Should the comparison be at the level of neurons (e.g. number of neurons in each layer), or at the level of weights (e.g. number of parameters for training in each type of network), or maybe something else?</p>
  
  <p>One idea that emerged was to construct one layer for the MLP for each corresponding convolutional layer, based on the number of neurons after the pooling and dropout layers.</p>
  
  <p>Any ideas? If there's any relative work or paper regarding this problem I would be very grateful to know.</p>
  
  <p>Thank you for your time</p>
  
  <p>Konstantinos</p>
  
Tags: <neural-networks><convolutional-neural-networks><comparison>
Title: How to make a fair comparison of a convolutional neural network (cNN) vs a mutlilayer perceptron (MLP)?
UserDisplayName: Constantine Pat.
ThreadText: # How to make a fair comparison of a convolutional neural network (cNN) vs a mutlilayer perceptron (MLP)?
  
  (Asked by: username_0 on 2018-06-08)
  
  I'm working with deep learning on some EEG data for classification, and I was wondering if there's any systematic/mathematical way to define the architecture of the networks, in order to compare their performance fairly.
  
  Should the comparison be at the level of neurons (e.g. number of neurons in each layer), or at the level of weights (e.g. number of parameters for training in each type of network), or maybe something else?
  
  One idea that emerged was to construct one layer for the MLP for each corresponding convolutional layer, based on the number of neurons after the pooling and dropout layers.
  
  Any ideas? If there's any relative work or paper regarding this problem I would be very grateful to know.
  
  Thank you for your time
  
  Konstantinos
  
  **Tags:** neural-networks, convolutional-neural-networks, comparison
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-06-13. Score: 0 | Accepted answer
  
  Konstantine, I assume you refer to plain MLP and CNN, without any modifications.
  
  I believe what you ask is how to set both of them up, in order to have the fairest comparison possible.
  
  The way I would do it, is to use their **plain implementations** but **both tuned as much as possible, in every hyperparameter**. Both should work as black boxes that accept the same inputs and give the same outputs.
  
  This will give you insight on the true raw performance of both algorithms.
  
  Hope it helps :)
  
  > **Comments:**
  > **username_0**: I'm not sure what plain implies. My models are sequential models with two hidden layers (for the convolutional network the first hidden layer is composed by two convolutional layers and a pooling layer). The idea of comparing the best out of the two is appealing. Although i don't have a lot of hyperparameters to play with, so I was thinking more of a size comparisons between the two (size in number of neurons, weights, parameters to learn, etc)  
  > **username_1**: By "plain" I refer to the most "basic" implementation of both algorithms. I don't see direct evaluation from just comparing the e.g. weights of both. This is because the architecture is different so you are comparing "apples and oranges". What would make sense is to compare the overall performance, not the inner characteristics. This means, don't focus on comparing hyperparameters, just make sure you fine-tune both networks. You should compare their results for the same inputs. Results like accuracy and training/inference time. Hope it makes sense. Kalh tyxh :)  
  
  ---
  
  ### Answer by username_2 on 2018-06-12. Score: 0
  
  The best best way to monitor an architectures performance would be comparing the resource utilization, model accuracy, value loss and confusion matrix.
    
  e.g VGG16 consumes less system resources in comparison to inception V3
    
    
  There is an [article](https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/) that goes in depth.
  
  > **Comments:**
  > **username_0**: The article talks more about various metrics of performance, which is not really my concern. The idea of resource utilization is interesting. Any ideas how to compare these in my models? (I use the keras library). Perhaps check the number of parameters to train for each network? (keras outputs the total number of trainable parameters)  
  
  ---
  
  