Answers.Body: <p>This is a great question because indeed there are many tools out there to make this part of the process faster.
  I have used as I usually stick to the following two:</p>
  <ul>
  <li><a href="https://pandas-profiling.github.io/pandas-profiling/docs/master/rtd/" rel="nofollow noreferrer">Pandas profiling</a></li>
  <li><a href="https://pair-code.github.io/facets/" rel="nofollow noreferrer">Facets</a></li>
  </ul>
  <p>You can also search for <a href="https://www.libhunt.com/r/pandas-profiling" rel="nofollow noreferrer">alternatives</a>
  Hopefully, the community can help complement my answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T22:58:36.243Z
Answers.LastActivityDate: 2021-08-03T22:58:36.243Z
Answers.UserDisplayName: Echo9k
Answers.Body: <p>all you mentioned belongs to any statistical analysis - it is suited for Normal Data &amp; is done with least-squares methods, but Maximum-Likelihood-Estimations can be applied for non-normal data (e.g. for rare-events analysis)... ML, in general, do not need Normality of data distribution, but some preprocessing is needed from the developer by his desire - see opportunities for preprocessing in <a href="https://scikit-learn.org/stable/modules/preprocessing" rel="nofollow noreferrer">sklearn's preprocessing class</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-02T07:54:04.190Z
Answers.LastActivityDate: 2024-06-02T07:54:04.190Z
Answers.UserDisplayName: JeeyCi
Body: <p>I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the distribution of each feature to see how normal it is, identify outliers, etc. Another example is plotting heatmaps of the features against themselves to identify strong correlations.</p>
  <p>Whilst this has been a useful learning exercise, going forward I suspect there are tools that automate a lot of this data analysis for you, and produce the useful plots and possibly give recommendations on transforms, etc? I've had a search around but don't appear to be finding anything, I guess I'm not using the right terminology to find what I'm looking for.</p>
  <p>If any useful open source tools for this kind of thing spring to mind that would be very helpful.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-04T10:25:43.923Z
Comments.Text: Hello. Welcome to Artificial Intelligence SE. It seems to me that this question is off-topic here and would be better suited for https://softwarerecs.stackexchange.com/ and https://datascience.stackexchange.com/. Asking for tools or software libraries is generally off-topic here. You can read more about our scope on [our on-topic page](https://ai.stackexchange.com/help/on-topic) (basically, we focus on the theoretical, philosophical and social aspects of AI, so programming issues, including asking for tools, are generally excluded).
Comments.UserDisplayName: nbro
Tags: <machine-learning><python><datasets><data-preprocessing><r>
Title: Data analysis before feeding to ML pipeline
UserDisplayName: Cosmicnet
ThreadText: # Data analysis before feeding to ML pipeline
  
  (Asked by: username_0 on 2021-08-03)
  
  I'm new to machine learning and I've been working through a dataset of ~3000 records with ~100 features. I've been hand rolling Python and R scripts to analyse the data. For example, plotting the distribution of each feature to see how normal it is, identify outliers, etc. Another example is plotting heatmaps of the features against themselves to identify strong correlations.
  
  Whilst this has been a useful learning exercise, going forward I suspect there are tools that automate a lot of this data analysis for you, and produce the useful plots and possibly give recommendations on transforms, etc? I've had a search around but don't appear to be finding anything, I guess I'm not using the right terminology to find what I'm looking for.
  
  If any useful open source tools for this kind of thing spring to mind that would be very helpful.
  
  **Tags:** machine-learning, python, datasets, data-preprocessing, r
  > **Comments:**
  > **username_1**: Hello. Welcome to Artificial Intelligence SE. It seems to me that this question is off-topic here and would be better suited for https://softwarerecs.stackexchange.com/ and https://datascience.stackexchange.com/. Asking for tools or software libraries is generally off-topic here. You can read more about our scope on [our on-topic page](https://ai.stackexchange.com/help/on-topic) (basically, we focus on the theoretical, philosophical and social aspects of AI, so programming issues, including asking for tools, are generally excluded).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-03. Score: 1 | Accepted answer
  
  This is a great question because indeed there are many tools out there to make this part of the process faster.
  I have used as I usually stick to the following two:
  
  * [Pandas profiling](https://pandas-profiling.github.io/pandas-profiling/docs/master/rtd/)
  * [Facets](https://pair-code.github.io/facets/)
  
  You can also search for [alternatives](https://www.libhunt.com/r/pandas-profiling)
  Hopefully, the community can help complement my answer.
  
  
  ---
  
  ### Answer by username_3 on 2024-06-02. Score: 0
  
  all you mentioned belongs to any statistical analysis - it is suited for Normal Data & is done with least-squares methods, but Maximum-Likelihood-Estimations can be applied for non-normal data (e.g. for rare-events analysis)... ML, in general, do not need Normality of data distribution, but some preprocessing is needed from the developer by his desire - see opportunities for preprocessing in [sklearn's preprocessing class](https://scikit-learn.org/stable/modules/preprocessing)
  
  
  ---
  
  