Answers.Body: <p>This is not a simple answer, and I think it really depends the goal, <em>quality</em> is not clearly defined and can vary. However there are some points that are commonly seen as positive.</p>
  <h1>The quantity</h1>
  <p>As you said, having enough data is important to train models, and it is even better if the dataset is balanced for classification. This means that it i spossible to build more complex models, and then <em>learn more</em> from the dataset.</p>
  <p><em>Learn more</em> does not mean to learn from more data, but from more <strong>diverse</strong> data. If you take the example of dog/cat classification, a larger dataset will probably have more angles (from above, behind or with an obscured view), and more cat/dog races which means that the algorithm can learn all these differences that needs a larger amount of data.</p>
  <p>The main point of quantity is not how much data, but how much of the input space is covered.</p>
  <h1>A clean dataset</h1>
  <p>The first step of every project is often cleaning the data and preprocessing it, which can be avoided if the dataset is of better &quot;quality&quot;. This means that all instances are correctly labeled, there is no missing value, no duplicates, the data types are the same...</p>
  <p>There the data is synthetic and can be used nearly as is, without need of further work to process these.</p>
  <h1>The ambiguous points</h1>
  <p>If a dataset present the attributes above I'd say it's quality is already great, but there are some additional points that can be considered. Some attributes of the dataset can vary (for example image size/resolution, background noise for audios, typo in language corpus...).</p>
  <p>The quality of the instances with such difference can be said to be low, if example if for example only the tail of the dog can be seen. However the issue is that <strong>real world data is not perfect</strong>.</p>
  <p>In some cases you can pre-process these differences or perturbations and correct them, but sometimes you can't. There some people will say it's part of the challenge to deal with these, some will say it is useless.
  There it depends on your goal and subjective point of view to decide if these inconsistency is needed or more harmful.</p>
  <p>To add a more personal thought, I'd say that when building a production model it is important to keep these as they reflect the real use and the perturbations brought by real data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T13:16:54.660Z
Answers.LastActivityDate: 2022-07-06T13:16:54.660Z
Answers.UserDisplayName: theophile
Answers.Body: <p>For the dataset to be useful for the community it should be publicly available in a stable domain and establish what the test set is (if you can have a hidden one and make a webpage to evaluate the submissions that's that's better). It's also useful to have baseline performance numbers.</p>
  <p>The dataset doesn't have to be balanced. High-quality datasets are regarded as such if the label noise is small (note that annotators do fail), the data are not corrupted and care has been taken regarding potentially harmful biases. Of course, having the dataset checked and reviewed by experts in the field is desirable. Creating a good dataset involves lots of work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-03T17:59:48.473Z
Answers.LastActivityDate: 2022-12-03T17:59:48.473Z
Answers.UserDisplayName: Franco Marchesoni
Body: <p>for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.</p>
  <p>While datasets from external sources are of course marketed as beeing 'good' or 'high-quality' (and in most cases not explained on how the authors come to this conclusion), creating a dataset yourself doesn't come with these labels.</p>
  <p>This brings me to my questions: <em>How can one (objectively) quantify the quality of a dataset for a given problem?</em></p>
  <p>This, of course, includes some points which are more or less 'accepted' within the community, e.g. the dataset has to contain enough datapoints for every modeled state (which leads to the question of what is 'enough' ...), or that the datapoints for each modeled state should be roughly equally split (e.g. a dataset for cat/dog image discrimination would not work well with one dog image and 10k cat images) and so on.</p>
  <p>I recognize that this is a rather open ended and maybe even philosophical question, but I believe that, given the importance of data for ML (and other disciplines), I am in need of an objective way to evaluate my datasets in relation to the task at hand and determine their quality in an objective way. Also my goal here is clearly ML oriented, but since this topic is not only valid in the ML context (and ML is in its core more or less complex statistics), I don't want to restrict this only to the ML, but to datasets overall</p>
  
Tags: <datasets><data-preprocessing>
Title: What makes a 'good' dataset
UserDisplayName: JakobS
ThreadText: # What makes a 'good' dataset
  
  (Asked by: username_0 on 2022-07-05)
  
  for the usage of ML technologies, having a appropriate dataset is arguably the first and fundamental step one has to tackle by either aquiring a dataset from external sources or creating their own.
  
  While datasets from external sources are of course marketed as beeing 'good' or 'high-quality' (and in most cases not explained on how the authors come to this conclusion), creating a dataset yourself doesn't come with these labels.
  
  This brings me to my questions: *How can one (objectively) quantify the quality of a dataset for a given problem?*
  
  This, of course, includes some points which are more or less 'accepted' within the community, e.g. the dataset has to contain enough datapoints for every modeled state (which leads to the question of what is 'enough' ...), or that the datapoints for each modeled state should be roughly equally split (e.g. a dataset for cat/dog image discrimination would not work well with one dog image and 10k cat images) and so on.
  
  I recognize that this is a rather open ended and maybe even philosophical question, but I believe that, given the importance of data for ML (and other disciplines), I am in need of an objective way to evaluate my datasets in relation to the task at hand and determine their quality in an objective way. Also my goal here is clearly ML oriented, but since this topic is not only valid in the ML context (and ML is in its core more or less complex statistics), I don't want to restrict this only to the ML, but to datasets overall
  
  **Tags:** datasets, data-preprocessing
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-07-06. Score: 1
  
  This is not a simple answer, and I think it really depends the goal, *quality* is not clearly defined and can vary. However there are some points that are commonly seen as positive.
  
  # The quantity
  
  As you said, having enough data is important to train models, and it is even better if the dataset is balanced for classification. This means that it i spossible to build more complex models, and then *learn more* from the dataset.
  
  *Learn more* does not mean to learn from more data, but from more **diverse** data. If you take the example of dog/cat classification, a larger dataset will probably have more angles (from above, behind or with an obscured view), and more cat/dog races which means that the algorithm can learn all these differences that needs a larger amount of data.
  
  The main point of quantity is not how much data, but how much of the input space is covered.
  
  # A clean dataset
  
  The first step of every project is often cleaning the data and preprocessing it, which can be avoided if the dataset is of better "quality". This means that all instances are correctly labeled, there is no missing value, no duplicates, the data types are the same...
  
  There the data is synthetic and can be used nearly as is, without need of further work to process these.
  
  # The ambiguous points
  
  If a dataset present the attributes above I'd say it's quality is already great, but there are some additional points that can be considered. Some attributes of the dataset can vary (for example image size/resolution, background noise for audios, typo in language corpus...).
  
  The quality of the instances with such difference can be said to be low, if example if for example only the tail of the dog can be seen. However the issue is that **real world data is not perfect**.
  
  In some cases you can pre-process these differences or perturbations and correct them, but sometimes you can't. There some people will say it's part of the challenge to deal with these, some will say it is useless.
  There it depends on your goal and subjective point of view to decide if these inconsistency is needed or more harmful.
  
  To add a more personal thought, I'd say that when building a production model it is important to keep these as they reflect the real use and the perturbations brought by real data.
  
  
  ---
  
  ### Answer by username_2 on 2022-12-03. Score: 0
  
  For the dataset to be useful for the community it should be publicly available in a stable domain and establish what the test set is (if you can have a hidden one and make a webpage to evaluate the submissions that's that's better). It's also useful to have baseline performance numbers.
  
  The dataset doesn't have to be balanced. High-quality datasets are regarded as such if the label noise is small (note that annotators do fail), the data are not corrupted and care has been taken regarding potentially harmful biases. Of course, having the dataset checked and reviewed by experts in the field is desirable. Creating a good dataset involves lots of work.
  
  
  ---
  
  