Answers.Body: <p>Goodness is subjective.  Reliable knowledge isn't possible with that flimsy a quality objective.</p>
  
  <p>The sturdy objective criteria you gave is 95%, so it is bad by that criteria.  (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)</p>
  
  <p>However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy.  (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.) </p>
  
  <p>I have four questions.  (My apologies that this question leads to the two assumptions above and a bunch more questions.)</p>
  
  <ul>
  <li>Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?</li>
  <li>Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?</li>
  <li>Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?</li>
  <li>What economic, contractual, or operational hard stop is dictating the 90%?</li>
  </ul>
  
  <p>If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-24T01:14:08.630Z
Answers.LastActivityDate: 2018-06-24T01:14:08.630Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I've selected more than 10 discriminative (classification) models, each wrapped with a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html" rel="nofollow noreferrer"><code>BaggingClassifier</code></a> object, optimized with a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html" rel="nofollow noreferrer"><code>GridSearchCV</code></a>, and all of them placed within a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html" rel="nofollow noreferrer"><code>VotingClassifier</code></a> object.</p>
  <p>Alone, they all bring around 70% accuracy, on a data set which is about half normal/uniform distributed, and half one-hot distributed. Together, they provide 80% accuracy, which isn't good enough, given that I was told that 95%&lt; is achievable.</p>
  <p>The models: <code>DecisionTreeClassifier</code>, <code>ExtraTreesClassifier</code>, <code>KNeighborsClassifier</code>, <code>GradientBoostingClassifier</code>, <code>LogisticRegression</code>, <code>SVC</code>, <code>Perceptron</code>, and a few more classifiers.</p>
  <p>How do I check if the combination is good?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T19:21:51.323Z
Comments.Text: Yeah, you can’t. If it works well it’s good. It’s a fairly odd combination tbh
Comments.UserDisplayName: Andreas Storvik Strauman
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T20:28:07.137Z
Comments.Text: The accuracy value can only be compared with other models. Having a goal is good, but unless you have a strong indication that your goal is achievable given the data in question, then just quoting a raw value like 70% or 99.9% is meaningless. How do human experts do given the same raw data?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-23T21:21:45.193Z
Comments.Text: Neil - It's for a HW assignment, and they claim 95%< is possible, probably too many features for a human to do 95%< .
  Andreas - I just took all the models I've found, with 70%< accuracy.
  What would be a good way to choose models ? and how many are probably needed (18 features data, half normal/uniform half one-hots) ?
Comments.UserDisplayName: Miko Diko
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-24T17:28:35.883Z
Comments.Text: OK, so if you trust your course materials, and 95% has been done, then you are missing something major. Were you expected to make a large ensemble as part of the task? I think you may be expected to do some feature engineering too - what course materials have you studied, and have you been told which ones are relevant to the task?
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><classification><performance><ensemble-learning><testing>
Title: How do I check that the combination of these models is good?
UserDisplayName: Miko Diko
ThreadText: # How do I check that the combination of these models is good?
  
  (Asked by: username_0 on 2018-06-23)
  
  I've selected more than 10 discriminative (classification) models, each wrapped with a [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) object, optimized with a [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html), and all of them placed within a [`VotingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html) object.
  
  Alone, they all bring around 70% accuracy, on a data set which is about half normal/uniform distributed, and half one-hot distributed. Together, they provide 80% accuracy, which isn't good enough, given that I was told that 95%< is achievable.
  
  The models: `DecisionTreeClassifier`, `ExtraTreesClassifier`, `KNeighborsClassifier`, `GradientBoostingClassifier`, `LogisticRegression`, `SVC`, `Perceptron`, and a few more classifiers.
  
  How do I check if the combination is good?
  
  **Tags:** machine-learning, classification, performance, ensemble-learning, testing
  > **Comments:**
  > **username_1**: Yeah, you can’t. If it works well it’s good. It’s a fairly odd combination tbh  
  > **username_2**: The accuracy value can only be compared with other models. Having a goal is good, but unless you have a strong indication that your goal is achievable given the data in question, then just quoting a raw value like 70% or 99.9% is meaningless. How do human experts do given the same raw data?  
  > **username_0**: Neil - It's for a HW assignment, and they claim 95%< is possible, probably too many features for a human to do 95%< . Andreas - I just took all the models I've found, with 70%< accuracy. What would be a good way to choose models ? and how many are probably needed (18 features data, half normal/uniform half one-hots) ?  
  > **username_2**: OK, so if you trust your course materials, and 95% has been done, then you are missing something major. Were you expected to make a large ensemble as part of the task? I think you may be expected to do some feature engineering too - what course materials have you studied, and have you been told which ones are relevant to the task?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2018-06-24. Score: 2
  
  Goodness is subjective. Reliable knowledge isn't possible with that flimsy a quality objective.
  
  The sturdy objective criteria you gave is 95%, so it is bad by that criteria. (I'm assuming that the 95% is expected for a given data set or a randomized sample from a given data set.)
  
  However, the 80% accuracy is good by the criteria where the you sum the measures of the accuracy of the individual models, divide by the number of models, and find you have gained ten percentage points of accuracy over that average with your aggregated execution strategy. (I'm assuming here that you used a defined set of network meta-parameters, layers depths and widths, starting parameters, activation model mapping to layers, inter-network connectivity, and loss/error methods for each model that is similar to the aggregated execution strategy.)
  
  I have four questions. (My apologies that this question leads to the two assumptions above and a bunch more questions.)
  
  * Is the 80% accuracy also well over the maximum of the accuracies of the set of accuracies from the individual models?
  * Is the computing resource draw to achieve 80% accuracy achievable with additional run time of the best of the individual models, using less than or equal to the computing resource draw to achieve 80% that way?
  * Have you run your evaluation with a full set of meta parameter vectors to check the entire meta-space for best case?
  * What economic, contractual, or operational hard stop is dictating the 90%?
  
  If we know these answers, we may be able to respond more effectively and possibly find a loop hole in the logic that appears to leave you with an undesirable foregone conclusion.
  
  
  ---
  
  