Body: <p>We have a scenario where we need to implement an Artificial Intelligence solution which will evaluate the input data file of my Azure Data Factory pipeline and let us know whether the file is good or bad with respect to it's data. </p>
  
  <p>For example, I have 10 files several rows which are good input files and 2 files with several rows which are bad input files.</p>
  
  <p>Each file either it is good/bad has 26 columns. The above two files are bad because of below reasons. </p>
  
  <ol>
  <li><p>One file has all empty values for one column which is not expected.</p></li>
  <li><p>Another file has, the value 'TRUE for all rows for a specific column which was also not he general scenario. (some % of TRUE's and some % records with FALSE will appear in good files)</p></li>
  </ol>
  
  <p>Like this, there may be several scenarios where the input file may be treated as bad file. </p>
  
  <p>We want to implement an Artificial Intelligence solution which should analyze all the input files and identify the hidden patterns of the data in file and detect abnormal scenarios like above and should eventually mark the file as bad file. </p>
  
  <p>Please suggest for the approach or what components in Azure can help to achieve this kind of file sanity check.</p>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T02:58:00.180Z
Comments.Text: This does sound a little bit like over-engineering. You might get a better result following predefined heuristics than implementing some form of machine learning.  That begin said, if you are set on using an AI, you could try using a feedfoward network (as this doesn't sound like a difficult task) and using % of TRUE's per column as an input, or other various values, with 2 output nodes of good or bad.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T03:11:53.097Z
Comments.Text: Thanks Recessive. Even i thought of doing with the help of file statistics like, type of column, how many rows has values, how many distinct values for each column, how many rows for each categorical ..etc. But, unfortunately, that was not accepted.BTW, I am new to machine learning/ AI. Could you please more details on your possible solution so  that I can search and understand it more.
Comments.UserDisplayName: Dileep Mittapalli
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T03:22:45.367Z
Comments.Text: You mightn't even need to use tensorflow or large libraries, try this: https://towardsdatascience.com/lets-code-a-neural-network-in-plain-numpy-ae7e74410795. If you do want to use a dedicated framework, here's a simple tutorial: https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T03:30:38.080Z
Comments.Text: Thank you very much Recessive. I will go-through them. Thanks for your immediate help.
Comments.UserDisplayName: Dileep Mittapalli
Tags: <machine-learning>
Title: Need to analyze input CSV files and determine whether input file is good or bad w.r.t it's data
UserDisplayName: Dileep Mittapalli
ThreadText: # Need to analyze input CSV files and determine whether input file is good or bad w.r.t it's data
  
  (Asked by: username_0 on 2019-12-10)
  
  We have a scenario where we need to implement an Artificial Intelligence solution which will evaluate the input data file of my Azure Data Factory pipeline and let us know whether the file is good or bad with respect to it's data.
  
  For example, I have 10 files several rows which are good input files and 2 files with several rows which are bad input files.
  
  Each file either it is good/bad has 26 columns. The above two files are bad because of below reasons.
  
  1. One file has all empty values for one column which is not expected.
  2. Another file has, the value 'TRUE for all rows for a specific column which was also not he general scenario. (some % of TRUE's and some % records with FALSE will appear in good files)
  
  Like this, there may be several scenarios where the input file may be treated as bad file.
  
  We want to implement an Artificial Intelligence solution which should analyze all the input files and identify the hidden patterns of the data in file and detect abnormal scenarios like above and should eventually mark the file as bad file.
  
  Please suggest for the approach or what components in Azure can help to achieve this kind of file sanity check.
  
  Thanks.
  
  **Tags:** machine-learning
  
  > **Comments:**
  > **username_1**: This does sound a little bit like over-engineering. You might get a better result following predefined heuristics than implementing some form of machine learning.  That begin said, if you are set on using an AI, you could try using a feedfoward network (as this doesn't sound like a difficult task) and using % of TRUE's per column as an input, or other various values, with 2 output nodes of good or bad.  
  > **username_0**: Thanks username_1. Even i thought of doing with the help of file statistics like, type of column, how many rows has values, how many distinct values for each column, how many rows for each categorical ..etc. But, unfortunately, that was not accepted.BTW, I am new to machine learning/ AI. Could you please more details on your possible solution so  that I can search and understand it more.  
  > **username_1**: You mightn't even need to use tensorflow or large libraries, try this: https://towardsdatascience.com/lets-code-a-neural-network-in-plain-numpy-ae7e74410795. If you do want to use a dedicated framework, here's a simple tutorial: https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/  
  > **username_0**: Thank you very much username_1. I will go-through them. Thanks for your immediate help.  
  
  ---
  
  ## 0 answers
  
  