Answers.Body: <p>The comments are off base.  Having worked in validation of data as a consultant for Nasdac, Amex, and Lexis Nexis I can tell you that using the UNIX sed -r or pcrelib is insufficient to do a stellar cleansing of address data.</p>
  
  <p>Although none of those companies did this at the time I was consulting, the application of current machine learning is easy to infer from the basic characteristics of data.  What is needed is a good record of incoming data, cleaned data, and rejected records to constantly use as a reference.</p>
  
  <p>The manual process would be a nightmare.  For instance, misspellings are more indicative of true data than properly spelled ones in some but not all cases (unless the falsified data generator is programmed to synthesize the existing distribution of misspellings in authentic data sources).</p>
  
  <p>Try to profile that in static code such that the data validation will adapt to data trends without requiring a maintenance workforce.  Those kinds of things never get addressed in a typical IT environment.  Non-adaptive validation would be a gross mistake, especially if the volume is high, like for M16 or Liberty Cross or the NSA or Interpol.</p>
  
  <p>What you want is an extremely fast index of well authenticated good addresses (perhaps corroborated by public records with local authentication policies) and a feature extraction from it using auto encoding or some similar and perhaps better methodology and its associated algorithms.</p>
  
  <p>Then you can train the classification of fake and authentic addresses based on extracted feature profiles.</p>
  
  <p>You will also want reinforcement, because the attackers (those trying to create false identities using plausible addresses) will adapt to the training of the current system.  The authentication system must stay steps ahead of those trying to defeat it, which they will likely try to do once the existence of authentication automation is detected.</p>
  
  <p>One can fend of attackers by placing misinformation strategically and then tracing the input sources back to the attacker based on the misinformation seeds.  That only works if you have law enforcement in your camp.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-28T01:52:02.933Z
Answers.LastActivityDate: 2018-07-28T02:02:14.650Z
Answers.LastEditDate: 2018-07-28T02:02:14.650Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I want to use a machine learning algorithm to detect false address data. I learned about neural networks and machine learning at university, but I don't have much experience in this field.</p>
  
  <p>Do you think it is feasible to use a high level algorithm for this or should I use simple queries and filters to catch out wrong data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-27T15:11:06.583Z
Comments.Text: Can you manually do the same task?
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-27T15:43:36.797Z
Comments.Text: What do you mean by "false"? Data that is not actually in the right format for an address, or addresses that are wrong for a particular location or person?
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-27T17:17:31.803Z
Comments.Text: - Yes I can do the task manually
Comments.UserDisplayName: Fatih Öz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-27T17:18:28.017Z
Comments.Text: - My mistake the data is sometimes not in the right format, sometimes its in the right format but there are spelling errors and sometimes the street is not in the written city.
Comments.UserDisplayName: Fatih Öz
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-27T20:55:14.320Z
Comments.Text: Well... Then you have to teach it the spelling rules of English. I'd honestly just use a regex, or check it against a database.
Comments.UserDisplayName: FreezePhoenix
Tags: <machine-learning>
Title: Machine learning to detect wrong address data
UserDisplayName: Fatih Öz
ThreadText: # Machine learning to detect wrong address data
  
  (Asked by: username_0 on 2018-07-27)
  
  I want to use a machine learning algorithm to detect false address data. I learned about neural networks and machine learning at university, but I don't have much experience in this field.
  
  Do you think it is feasible to use a high level algorithm for this or should I use simple queries and filters to catch out wrong data?
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: Can you manually do the same task?  
  > **username_2**: What do you mean by "false"? Data that is not actually in the right format for an address, or addresses that are wrong for a particular location or person?  
  > **username_0**: - Yes I can do the task manually  
  > **username_0**: - My mistake the data is sometimes not in the right format, sometimes its in the right format but there are spelling errors and sometimes the street is not in the written city.  
  > **username_3**: Well... Then you have to teach it the spelling rules of English. I'd honestly just use a regex, or check it against a database.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2018-07-28. Score: 1
  
  The comments are off base. Having worked in validation of data as a consultant for Nasdac, Amex, and Lexis Nexis I can tell you that using the UNIX sed -r or pcrelib is insufficient to do a stellar cleansing of address data.
  
  Although none of those companies did this at the time I was consulting, the application of current machine learning is easy to infer from the basic characteristics of data. What is needed is a good record of incoming data, cleaned data, and rejected records to constantly use as a reference.
  
  The manual process would be a nightmare. For instance, misspellings are more indicative of true data than properly spelled ones in some but not all cases (unless the falsified data generator is programmed to synthesize the existing distribution of misspellings in authentic data sources).
  
  Try to profile that in static code such that the data validation will adapt to data trends without requiring a maintenance workforce. Those kinds of things never get addressed in a typical IT environment. Non-adaptive validation would be a gross mistake, especially if the volume is high, like for M16 or Liberty Cross or the NSA or Interpol.
  
  What you want is an extremely fast index of well authenticated good addresses (perhaps corroborated by public records with local authentication policies) and a feature extraction from it using auto encoding or some similar and perhaps better methodology and its associated algorithms.
  
  Then you can train the classification of fake and authentic addresses based on extracted feature profiles.
  
  You will also want reinforcement, because the attackers (those trying to create false identities using plausible addresses) will adapt to the training of the current system. The authentication system must stay steps ahead of those trying to defeat it, which they will likely try to do once the existence of authentication automation is detected.
  
  One can fend of attackers by placing misinformation strategically and then tracing the input sources back to the attacker based on the misinformation seeds. That only works if you have law enforcement in your camp.
  
  
  ---
  
  