Answers.Body: <p>If I get correctly what you're asking, the answer is no, there is no way of knowing in advance how good a model would perform in a dataset without training a model on it. That's the whole point of data science, you try, you analyse the results, you try again using the knowledge you got from your previous attempts. It would be nice to hack the whole field and know in advance what to do to get the perfect model but it is rather unrealistic. </p>
  
  <p>Anyway, there are some standard steps that usually help understanding if you're going on the right direction. For example creating a random benchmark to see how much your model is better than a random one. To create such benchmark you definitely don't need a specific tool, all main programming languages provide built in function to generate random numbers, and that's basically all that you need. For example in python you could do something like:</p>
  
  <pre><code>import numpy as np
  
  # create 30 true labels 
  y_true = [1,2,3]*10
  
  # generate 30 random labels
  y_random = np.random.randint(0,3,size=30)
  
  # calculate accuracy for random model
  acc_random = sum(y_true==y_random)/len(y_true) 
  </code></pre>
  
  <p>Another thing that is worth to mention is that in the academy people are pushing more and more to use the same datasets in order to have comparable results. for example if you're trying to train an architecture that should be used for images classification, some golden dataset you must test your models on are the <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST</a> ones. It is well known that on these datasets a good model should achieve more than 99% accuracy, therefore this is an establish lower bound for these datasets. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T15:21:56.997Z
Answers.LastActivityDate: 2020-04-13T15:21:56.997Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>There is no tool to achieve what you desire. A "rough" way to get a guess is to "compare" your data set to other known data sets for which there are benchmark accuracy models. Compare is a vague definition at best but things you might compare are</p>
  
  <ol>
  <li>number of your classes versus the number of classes in the reference data set</li>
  <li>the number of train, validation and test samples</li>
  <li>the relative similarity of the data - that is are they images? If so image comapare size, are the images cropped to the region of interest, etc</li>
  <li>similarity of class features -that is are your classes very similar to each other. For example classifying dogs by breed can be difficult because some dog breeds look almost identical versus a situation like classifying different types of animals which is easier. Estimate your similarity of classes with respect to
  similarity of classes in the reference data set. </li>
  </ol>
  
  <p>So you might be able to generate a "rough" estimate of what you might expect your model to achieve particularly if you use transfer learning of benchmark models. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T17:26:09.147Z
Answers.LastActivityDate: 2020-04-13T17:26:09.147Z
Answers.UserDisplayName: Gerry P
Body: <p>Is there an online tool that can predict accuracy given only the dataset as input (i.e. without the compiled model)?</p>
  
  <p>That would help to understand how data augmentation/distribution standardization, etc., is likely to change the accuracy.</p>
  
Tags: <machine-learning><deep-learning><resource-request>
Title: Is there an online tool that can predict accuracy given only the dataset?
UserDisplayName: jaiswati_b
ThreadText: # Is there an online tool that can predict accuracy given only the dataset?
  
  (Asked by: username_0 on 2020-04-13)
  
  Is there an online tool that can predict accuracy given only the dataset as input (i.e. without the compiled model)?
  
  That would help to understand how data augmentation/distribution standardization, etc., is likely to change the accuracy.
  
  **Tags:** machine-learning, deep-learning, resource-request
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-13. Score: 1
  
  If I get correctly what you're asking, the answer is no, there is no way of knowing in advance how good a model would perform in a dataset without training a model on it. That's the whole point of data science, you try, you analyse the results, you try again using the knowledge you got from your previous attempts. It would be nice to hack the whole field and know in advance what to do to get the perfect model but it is rather unrealistic.
  
  Anyway, there are some standard steps that usually help understanding if you're going on the right direction. For example creating a random benchmark to see how much your model is better than a random one. To create such benchmark you definitely don't need a specific tool, all main programming languages provide built in function to generate random numbers, and that's basically all that you need. For example in python you could do something like:
  
  ```
  import numpy as np
  
  # create 30 true labels 
  y_true = [1,2,3]*10
  
  # generate 30 random labels
  y_random = np.random.randint(0,3,size=30)
  
  # calculate accuracy for random model
  acc_random = sum(y_true==y_random)/len(y_true) 
  
  ```
  
  Another thing that is worth to mention is that in the academy people are pushing more and more to use the same datasets in order to have comparable results. for example if you're trying to train an architecture that should be used for images classification, some golden dataset you must test your models on are the [MNIST](http://yann.lecun.com/exdb/mnist/) ones. It is well known that on these datasets a good model should achieve more than 99% accuracy, therefore this is an establish lower bound for these datasets.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-13. Score: 1
  
  There is no tool to achieve what you desire. A "rough" way to get a guess is to "compare" your data set to other known data sets for which there are benchmark accuracy models. Compare is a vague definition at best but things you might compare are
  
  1. number of your classes versus the number of classes in the reference data set
  2. the number of train, validation and test samples
  3. the relative similarity of the data - that is are they images? If so image comapare size, are the images cropped to the region of interest, etc
  4. similarity of class features -that is are your classes very similar to each other. For example classifying dogs by breed can be difficult because some dog breeds look almost identical versus a situation like classifying different types of animals which is easier. Estimate your similarity of classes with respect to
     similarity of classes in the reference data set.
  
  So you might be able to generate a "rough" estimate of what you might expect your model to achieve particularly if you use transfer learning of benchmark models.
  
  
  ---
  
  