Answers.Body: <p>Before jumping to modeling, there are a few tasks a data scientist (or ML/AI practitioner) must do:</p>
  
  <ol>
  <li><strong>Ideation</strong> (or hypothesizing): Before applying any modeling approach, we need to ask the right questions. We must clearly mention our assumptions and declare how we want to measure the effectiveness of the pipeline. Note that, some tools/algorithms might not fit to the made assumptions or may not lead to the best values in the defined metrics. So, the pipeline must be designed in a way it serves the purpose of answering the defined questions.</li>
  <li><a href="https://www.udemy.com/topic/data-cleaning/" rel="nofollow noreferrer"><strong>Data Cleansing</strong></a>: Real-world data sets are usually not clean. They have all sorts of data issues such as missing value, duplicates, outliers, wrong measurements, fragments, inconsistency, etc. Most of the ML techniques are sensitive (to different extents, of course) to such issues. Therefore, the data should be cleaned before any modeling can be done.</li>
  <li><strong>Data Wrangling</strong> (or Feature Engineering): In many cases, the gathered data (even cleaned) is not immediately suitable for any modeling/analysis. For example, we may need to convert the documents of a text corpus to vectors of numbers (via TF-IDF or Embedding techniques) before being able to apply a text classifier simply because our classifier only takes numeric data. Converting measurements to other units, breaking addresses to their components, converting times and dates to different formats or timezones are just a few examples of data wrangling tasks (in a broader context, feature engineering may also refer to dimension reduction or feature selection/projection).</li>
  <li><a href="https://www.udemy.com/exploratory-data-analysis-with-pandas-and-python-3x/" rel="nofollow noreferrer"><strong>Exploratory Data Analysis (EDA)</strong></a>: To do the cleansing, understanding the data set features, and ideation, we often need to explore the given data set using visual (e.g., dashboards and diagrams) or summary statistics tools.</li>
  </ol>
  
  <p><em>Disclaimer: I have no business interest with Udemy. The links are just shared because @pedro-de-sá mentioned they take some courses from Udemy.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T11:28:32.050Z
Answers.Text: I would add that data cleansing and feature engineering are often part of the _data preprocessing_ phase.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T11:12:28.680Z
Answers.Text: Thank you so much, @Borhan Kazimipour! Also if you have any books, articles or anything else to indicate me, works just fine too! It doesn't necessarily need to be Udemy's courses. Thanks once again!
Answers.UserDisplayName: Pedro de Sá
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T08:57:29.227Z
Answers.LastActivityDate: 2019-06-27T11:27:49.387Z
Answers.LastEditDate: 2019-06-27T11:27:49.387Z
Answers.UserDisplayName: Borhan Kazimipour
Body: <p>I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing.</p>
  
  <p>In all of those Udemy's lessons, people always use a perfect dataset and ready to input in a model. So all you have to do is run it.</p>
  
  <p><em>How do I know if my dataset is ready for a model?</em> <em>What do I have to do to make it ready? Which evaluations?</em></p>
  
  <p>I had a few statistics classes in college already and I learned a lot about correlations matrices, autocorrelation functions, and its lags, etc. and I didn't see yet in anywhere someone explaining how can I evaluate my data and then proceed to implement a model to solve my problem.</p>
  
  <p>If anyone could point me a direction, give me some material, show me where I can learn this, anything, it would be really helpful!</p>
  
Tags: <neural-networks><machine-learning><datasets><data-science><data-mining>
Title: How do I know if my dataset is ready for a machine learning model?
UserDisplayName: Pedro de Sá
ThreadText: # How do I know if my dataset is ready for a machine learning model?
  
  (Asked by: username_0 on 2019-06-27)
  
  I am new in this area of Machine Learning and Neural Networks. Currently, I'm taking some courses on Udemy and reading a book about it, but I still have one big question regarding data pre-processing.
  
  In all of those Udemy's lessons, people always use a perfect dataset and ready to input in a model. So all you have to do is run it.
  
  *How do I know if my dataset is ready for a model?* *What do I have to do to make it ready? Which evaluations?*
  
  I had a few statistics classes in college already and I learned a lot about correlations matrices, autocorrelation functions, and its lags, etc. and I didn't see yet in anywhere someone explaining how can I evaluate my data and then proceed to implement a model to solve my problem.
  
  If anyone could point me a direction, give me some material, show me where I can learn this, anything, it would be really helpful!
  
  **Tags:** neural-networks, machine-learning, datasets, data-science, data-mining
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-27. Score: 2 | Accepted answer
  
  Before jumping to modeling, there are a few tasks a data scientist (or ML/AI practitioner) must do:
  
  1. **Ideation** (or hypothesizing): Before applying any modeling approach, we need to ask the right questions. We must clearly mention our assumptions and declare how we want to measure the effectiveness of the pipeline. Note that, some tools/algorithms might not fit to the made assumptions or may not lead to the best values in the defined metrics. So, the pipeline must be designed in a way it serves the purpose of answering the defined questions.
  2. [**Data Cleansing**](https://www.udemy.com/topic/data-cleaning/): Real-world data sets are usually not clean. They have all sorts of data issues such as missing value, duplicates, outliers, wrong measurements, fragments, inconsistency, etc. Most of the ML techniques are sensitive (to different extents, of course) to such issues. Therefore, the data should be cleaned before any modeling can be done.
  3. **Data Wrangling** (or Feature Engineering): In many cases, the gathered data (even cleaned) is not immediately suitable for any modeling/analysis. For example, we may need to convert the documents of a text corpus to vectors of numbers (via TF-IDF or Embedding techniques) before being able to apply a text classifier simply because our classifier only takes numeric data. Converting measurements to other units, breaking addresses to their components, converting times and dates to different formats or timezones are just a few examples of data wrangling tasks (in a broader context, feature engineering may also refer to dimension reduction or feature selection/projection).
  4. [**Exploratory Data Analysis (EDA)**](https://www.udemy.com/exploratory-data-analysis-with-pandas-and-python-3x/): To do the cleansing, understanding the data set features, and ideation, we often need to explore the given data set using visual (e.g., dashboards and diagrams) or summary statistics tools.
  
  *Disclaimer: I have no business interest with Udemy. The links are just shared because @pedro-de-sá mentioned they take some courses from Udemy.*
  
  > **Comments:**
  > **username_2**: I would add that data cleansing and feature engineering are often part of the _data preprocessing_ phase.  
  > **username_0**: Thank you so much, @Borhan Kazimipour! Also if you have any books, articles or anything else to indicate me, works just fine too! It doesn't necessarily need to be Udemy's courses. Thanks once again!  
  
  ---
  
  