Answers.Body: <p>You would need to define 'frames', 'templates', or sets of data belonging together to form an address or other kind of data, with typical labels. So <strong>phone</strong> or <strong>tel</strong> etc would indicate a phone number, provided that their content also looks like a phone number. That's how you as a human recognise it. So you encode your domain knowledge as entities with possible attributes. Then you try and match the attributes and recognise which entity they belong to. You could have several entities with a shared subset of attributes (like a company or a person, which would both have an address). There would be other clues to tell you which it is. If <strong>name</strong> ends in "Ltd." or "Co", then it'll be a company, for example.</p>
  
  <p>So you mix heuristics for identifying attributes, templates of which attributes combine to form an entity class you want to identify, and then pick the one that best matches. If you have several entities where all that's filled is <strong>phone</strong>, then you can't really tell what it is and would discard it. In your example, <strong>name</strong> matches various entities, but <strong>office</strong> would not be a valid attribute for a person (unless you decided it was). An oversimplistic heuristic might think "Sam Smith" is a company, which presumably has an <strong>office</strong> attribute, so you need to be careful with how you design your templates.</p>
  
  <p>A criterion for putting attributes together could be that they are in the same subtree of the data structure. The exact definition really depends on the data and the types of information you want to extract from it.</p>
  
  <p>So yes, it is definitely possible. I'm not really sure which framework, but it should be fairly trivial to code this in a programming language of your choice.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-04T19:33:47.763Z
Answers.LastActivityDate: 2017-09-04T19:33:47.763Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>XML, HTML and less formal languages all respond quite nicely to being transformed or interrogated within a graph framework. XML and HTML are particularly useful in that they conform strictly to a tree-structure. That means that any good data components can be measured in terms of tree-distance to any other "good" data components. </p>
  
  <p>If you extract your regex-friendly terms and keep track of where within their tree they are found, you <em>may</em> be able to cast those values into a general document-space vector (it might only need to be one-dimensional), allowing you to identify clusters of "good" vs anomalous sections of "bad" data based on a simple distance metric, or anomaly detection algorithm - say an <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html" rel="noreferrer">isolation-forest</a> that runs on information density for example. </p>
  
  <p>This depends on your data, and how much of it you can find, ideally already tagged up containing good vs bad. </p>
  
  <p>If you're looking to scrape reliable address-contents, then yes, you're likely to score hits on names, addresses, postcodes and phone-numbers all appearing as tightly connected clustered groups, all within one or two nodes-distance from one another.</p>
  
  <p>Meanwhile, an annotation containing a phone number lodged somewhere else is less likely to be a match. </p>
  
  <p>Different documents will have different threshold densities, and differing anomaly to conformant ratios, so you'd have a task on your hands to figure out some way to automatically tune your parameters on any given document set. </p>
  
  <p>In the past, I've tried doing this against html by flattening all the content into a single string of text and a similar approach yielded half-decent results, but if you're looking at XML, it's fair to expect the structure to yield more information. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-30T12:41:52.603Z
Answers.Text: Thanks for the answer.Do you have any good graph frameworks as starting point? I'm pretty new to machine learning, so a few pointers into the right direction would be highly appreciated.
Answers.UserDisplayName: DBX12
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-30T14:34:15.547Z
Answers.Text: I use python a lot for things like this, and [networkx](https://networkx.github.io/) is a fantastic graph-framework. I might use of of the xml parsing modules discussed [here](https://stackoverflow.com/questions/1912434/how-do-i-parse-xml-in-python) to parse and marshal the content, perhaps pushing that into a networkx graph for the added analytical flexibility that might give you.
Answers.UserDisplayName: Thomas Kimber
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-04T17:45:16.163Z
Answers.LastActivityDate: 2017-09-04T17:45:16.163Z
Answers.UserDisplayName: Thomas Kimber
Body: <p>I think about a system which gets XML documents in various structures but with essentially the same data structure in it. For the example, let's assume each document contains data about one or more persons. So the AI would recognize a name. Somewhere else in the document there is the post address of our fictional person. The AI should now &quot;see&quot; the address and conclude, it belongs to our person. Anywhere else, there is a phone number in the document. Again, our AI should see the connection between our person and this phone number.</p>
  <p>This wouldn't be a job for an AI if there wasn't a catch. If the task was merely to find and map strings like addresses and phone numbers, we could simply use a regex to match our &quot;target strings&quot;. The catch in this scenario would be this: the XML document might contain other data, which does not belong to our person but is a valid phone number for example and thus will match an regex.</p>
  <p><em>Would it be possible for an AI to learn this? If yes, with which framework would someone create such an AI?</em></p>
  <hr />
  <p>Sample XML document:</p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot; ?&gt;
  &lt;document&gt;
      &lt;data&gt;
          &lt;foo&gt;
              &lt;bar&gt;
                  &lt;person&gt;
                      &lt;name&gt;John Doe&lt;/name&gt;
                  &lt;/person&gt;
              &lt;/bar&gt;
              &lt;address&gt;
                  &lt;street&gt;Main street 1&lt;/street&gt;
                  &lt;city&gt;1111 Twilight town&lt;/city&gt;
                  &lt;country&gt;sample country&lt;/country&gt;
              &lt;/address&gt;
              &lt;phone&gt;+123 123 123&lt;/phone&gt;
          &lt;/foo&gt;
          &lt;foo&gt;
              &lt;bar&gt;
                  &lt;person&gt;
                      &lt;name&gt;Jane Doe&lt;/name&gt;
                  &lt;/person&gt;
              &lt;/bar&gt;
              &lt;address&gt;
                  &lt;street&gt;Broadway 42&lt;/street&gt;
                  &lt;city&gt;4521 Traverse town&lt;/city&gt;
                  &lt;country&gt;sample country&lt;/country&gt;
              &lt;/address&gt;
              &lt;phone&gt;+123 412123&lt;/phone&gt;
          &lt;/foo&gt;
      &lt;/data&gt;
      &lt;creator&gt;
          &lt;!-- Note: While this looks like a valid person, --&gt;
          &lt;!-- this data should not be matched by the AI --&gt;
          &lt;name&gt;Sam Smith&lt;/name&gt;
          &lt;office&gt;
              &lt;street&gt;Seaside road 5&lt;/street&gt;
              &lt;city&gt;4521 Traverse town&lt;/city&gt;
              &lt;country&gt;sample country&lt;/country&gt;
          &lt;/office&gt;
          &lt;phone&gt;+123 555 555&lt;/phone&gt;
      &lt;/creator&gt;
  &lt;/document&gt;
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-01T18:08:02.680Z
Comments.Text: The first question I'd ask is "can a human make the distinction you're asking about, and how?"
Comments.UserDisplayName: mindcrime
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-01T18:27:35.073Z
Comments.Text: Welcome to AI!  *(I very much like mindcrime's comment, b/c I know how I'd go about validating the information, and suspect an AI could do it much more quickly.)*
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-01T18:55:29.693Z
Comments.Text: I'm guessing that what you really want is an AI that, given an arbitrary XML schema, can guess which elements/attributes the data you want is going to be encoded in.  So to train your AI, you would need a large number of different schemas with the correct elements already tagged.
Comments.UserDisplayName: antlersoft
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-02T11:19:47.807Z
Comments.Text: @mindcrime The human can make the distinction between, because the data belonging together would be more or less under the same parent. I updated my question with a sample XML. As you can see, the data about one person is below the `<foo>` parent. For humans it's clear that each `<foo>` represents one person.
Comments.UserDisplayName: DBX12
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-02T11:23:48.203Z
Comments.Text: @antlersoft That was my initial idea. I think I have enough XML schemas at hand (otherwise I will generate some via script from sample data). My line of thought was "Humans can learn to differentiate that. AI can learn like humans so AI should be able to learn this too". But then the question "how to start and are there frameworks?" rose, resulting in this question,
Comments.UserDisplayName: DBX12
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-02T11:26:52.743Z
Comments.Text: @DukeZhou Thank you for the welcome. I'm not sure I follow about the "AI could do it much more quickly" part. Do you think an AI wouldn't be faster than a human identifying the correct nodes?
Comments.UserDisplayName: DBX12
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-03T21:40:46.513Z
Comments.Text: @DBX12 I was meaning that an AI could go about validating the data, once it identified data needing to be validated, much more quickly than I could by hand (googling a phone number, for instance, to make sure it is associated with the person in question.)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-09-04T06:42:37.857Z
Comments.Text: @DukeZhou ah, now I understand. I added a sample of a false positive to my example code. In reality, almost all phone numbers are valid and the check if the phone is reachable should be no concern to the AI.
Comments.UserDisplayName: DBX12
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-07T12:09:05.353Z
Comments.Text: I am working on a platform which will apply deep learning on XML documents. Please check out my github repo [xml.ai](https://github.com/nishantsharma/xml.ai). This is a work in progress. Please add a star if you like.
Comments.UserDisplayName: Nishant 
Tags: <machine-learning><applications>
Title: Can I use AI to interpret XML documents?
UserDisplayName: DBX12
ThreadText: # Can I use AI to interpret XML documents?
  
  (Asked by: username_0 on 2017-09-01)
  
  I think about a system which gets XML documents in various structures but with essentially the same data structure in it. For the example, let's assume each document contains data about one or more persons. So the AI would recognize a name. Somewhere else in the document there is the post address of our fictional person. The AI should now "see" the address and conclude, it belongs to our person. Anywhere else, there is a phone number in the document. Again, our AI should see the connection between our person and this phone number.
  
  This wouldn't be a job for an AI if there wasn't a catch. If the task was merely to find and map strings like addresses and phone numbers, we could simply use a regex to match our "target strings". The catch in this scenario would be this: the XML document might contain other data, which does not belong to our person but is a valid phone number for example and thus will match an regex.
  
  *Would it be possible for an AI to learn this? If yes, with which framework would someone create such an AI?*
  
  ---
  
  Sample XML document:
  
  ```
  <?xml version="1.0" encoding="utf-8" ?>
  <document>
      <data>
          <foo>
              <bar>
                  <person>
                      <name>John Doe</name>
                  </person>
              </bar>
              <address>
                  <street>Main street 1</street>
                  <city>1111 Twilight town</city>
                  <country>sample country</country>
              </address>
              <phone>+123 123 123</phone>
          </foo>
          <foo>
              <bar>
                  <person>
                      <name>Jane Doe</name>
                  </person>
              </bar>
              <address>
                  <street>Broadway 42</street>
                  <city>4521 Traverse town</city>
                  <country>sample country</country>
              </address>
              <phone>+123 412123</phone>
          </foo>
      </data>
      <creator>
          <!-- Note: While this looks like a valid person, -->
          <!-- this data should not be matched by the AI -->
          <name>Sam Smith</name>
          <office>
              <street>Seaside road 5</street>
              <city>4521 Traverse town</city>
              <country>sample country</country>
          </office>
          <phone>+123 555 555</phone>
      </creator>
  </document>
  
  ```
  
  **Tags:** machine-learning, applications
  
  > **Comments:**
  > **username_1**: The first question I'd ask is "can a human make the distinction you're asking about, and how?"  
  > **username_2**: Welcome to AI!  *(I very much like username_1's comment, b/c I know how I'd go about validating the information, and suspect an AI could do it much more quickly.)*  
  > **username_3**: I'm guessing that what you really want is an AI that, given an arbitrary XML schema, can guess which elements/attributes the data you want is going to be encoded in.  So to train your AI, you would need a large number of different schemas with the correct elements already tagged.  
  > **username_0**: @username_1 The human can make the distinction between, because the data belonging together would be more or less under the same parent. I updated my question with a sample XML. As you can see, the data about one person is below the `<foo>` parent. For humans it's clear that each `<foo>` represents one person.  
  > **username_0**: @username_3 That was my initial idea. I think I have enough XML schemas at hand (otherwise I will generate some via script from sample data). My line of thought was "Humans can learn to differentiate that. AI can learn like humans so AI should be able to learn this too". But then the question "how to start and are there frameworks?" rose, resulting in this question,  
  > **username_0**: @username_2 Thank you for the welcome. I'm not sure I follow about the "AI could do it much more quickly" part. Do you think an AI wouldn't be faster than a human identifying the correct nodes?  
  > **username_2**: @username_0 I was meaning that an AI could go about validating the data, once it identified data needing to be validated, much more quickly than I could by hand (googling a phone number, for instance, to make sure it is associated with the person in question.)  
  > **username_0**: @username_2 ah, now I understand. I added a sample of a false positive to my example code. In reality, almost all phone numbers are valid and the check if the phone is reachable should be no concern to the AI.  
  > **username_4**: I am working on a platform which will apply deep learning on XML documents. Please check out my github repo [xml.ai](https://github.com/nishantsharma/xml.ai). This is a work in progress. Please add a star if you like.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_6 on 2017-09-04. Score: 6 | Accepted answer
  
  XML, HTML and less formal languages all respond quite nicely to being transformed or interrogated within a graph framework. XML and HTML are particularly useful in that they conform strictly to a tree-structure. That means that any good data components can be measured in terms of tree-distance to any other "good" data components.
  
  If you extract your regex-friendly terms and keep track of where within their tree they are found, you *may* be able to cast those values into a general document-space vector (it might only need to be one-dimensional), allowing you to identify clusters of "good" vs anomalous sections of "bad" data based on a simple distance metric, or anomaly detection algorithm - say an [isolation-forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html) that runs on information density for example.
  
  This depends on your data, and how much of it you can find, ideally already tagged up containing good vs bad.
  
  If you're looking to scrape reliable address-contents, then yes, you're likely to score hits on names, addresses, postcodes and phone-numbers all appearing as tightly connected clustered groups, all within one or two nodes-distance from one another.
  
  Meanwhile, an annotation containing a phone number lodged somewhere else is less likely to be a match.
  
  Different documents will have different threshold densities, and differing anomaly to conformant ratios, so you'd have a task on your hands to figure out some way to automatically tune your parameters on any given document set.
  
  In the past, I've tried doing this against html by flattening all the content into a single string of text and a similar approach yielded half-decent results, but if you're looking at XML, it's fair to expect the structure to yield more information.
  
  > **Comments:**
  > **username_0**: Thanks for the answer.Do you have any good graph frameworks as starting point? I'm pretty new to machine learning, so a few pointers into the right direction would be highly appreciated.  
  > **username_6**: I use python a lot for things like this, and [networkx](https://networkx.github.io/) is a fantastic graph-framework. I might use of of the xml parsing modules discussed [here](https://stackoverflow.com/questions/1912434/how-do-i-parse-xml-in-python) to parse and marshal the content, perhaps pushing that into a networkx graph for the added analytical flexibility that might give you.  
  
  ---
  
  ### Answer by username_5 on 2017-09-04. Score: 2
  
  You would need to define 'frames', 'templates', or sets of data belonging together to form an address or other kind of data, with typical labels. So **phone** or **tel** etc would indicate a phone number, provided that their content also looks like a phone number. That's how you as a human recognise it. So you encode your domain knowledge as entities with possible attributes. Then you try and match the attributes and recognise which entity they belong to. You could have several entities with a shared subset of attributes (like a company or a person, which would both have an address). There would be other clues to tell you which it is. If **name** ends in "Ltd." or "Co", then it'll be a company, for example.
  
  So you mix heuristics for identifying attributes, templates of which attributes combine to form an entity class you want to identify, and then pick the one that best matches. If you have several entities where all that's filled is **phone**, then you can't really tell what it is and would discard it. In your example, **name** matches various entities, but **office** would not be a valid attribute for a person (unless you decided it was). An oversimplistic heuristic might think "Sam Smith" is a company, which presumably has an **office** attribute, so you need to be careful with how you design your templates.
  
  A criterion for putting attributes together could be that they are in the same subtree of the data structure. The exact definition really depends on the data and the types of information you want to extract from it.
  
  So yes, it is definitely possible. I'm not really sure which framework, but it should be fairly trivial to code this in a programming language of your choice.
  
  
  ---
  
  