Answers.Body: <p>Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do <code>interface</code> and <code>Interface</code> mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called &quot;keywords&quot; of your configurations (such as <code>vlan</code>), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.</p>
  <p>Then write rules to replace &quot;arbitrary&quot; data (numbers, ips, ...) with a common tag (for example <code>&lt;NUMBER&gt;</code>, <code>&lt;IP&gt;</code>, ...). You should replace all non-keyword words (aka. tokens) with these tags. Use <code>&lt;?&gt;</code> for tokens you don't identify with a regex, or leave them as-is.</p>
  <p>Now we can calculate statistics of these token <a href="https://en.wikipedia.org/wiki/N-gram" rel="nofollow noreferrer">n-grams</a>, and identify those which occur very rarely. This step will require some thingking and tuning, for example how to handle newlines (or just ignore them?), how long n-grams to use, how rare is too rare etc.</p>
  <p>This is more of a heuristic than an AI, but it works without any training data. Well, there is a human in the loop so at least it is an Intelligent solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-07T20:33:39.137Z
Answers.Text: Nice idea @NikoNyrth.
  I'll give it a shot.
Answers.UserDisplayName: AJN
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T21:06:41.967Z
Answers.LastActivityDate: 2022-01-07T20:40:27.983Z
Answers.LastEditDate: 2022-01-07T20:40:27.983Z
Answers.UserDisplayName: NikoNyrh
Body: <p>Is there a way to use python and AI to compare thousands of files and detect the ones containing &quot;unusual&quot; content?</p>
  <p>Those files are supposed to have &quot;homogeneous&quot; configuration types with different parameters, but sometimes some of them might contain &quot;illegal&quot; types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.</p>
  <p>Here is one of too many bloc of configuration considered &quot;correct&quot;: (the meaning doesn't matter)</p>
  <pre><code>vlan XXXX
    name  XXX_NAME
  
  interface vxlan 1
    vxlan vlan XXX vni YYYY
  
  router bgp YYYY
     vlan XXXX
        rd A.B.C.D:YYYY
        route-target both YYYY:YYYY
        redistribute learned
  </code></pre>
  <p>Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example</p>
  <pre><code>Interface vlan AAAA
    ip address B.B.B.B/Z
  </code></pre>
  <p>Because it is in only a couple of files (&quot;rare&quot;) it is more likely that it is not authorized configuration.</p>
  <p>I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.</p>
  <p>I am wondering if AI could help with this issue?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-04T15:59:01.673Z
Comments.Text: It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T07:03:40.413Z
Comments.Text: What makes an illegal configuration illegal? How do you detect that?
Comments.UserDisplayName: AlDante
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-05T13:08:20.533Z
Comments.Text: Hey, what I call "illegal" is lines different from the most frequent ones (correct).
  I'll complete the question with an example.
Comments.UserDisplayName: AJN
Tags: <machine-learning><training><python>
Title: Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?
UserDisplayName: AJN
ThreadText: # Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?
  
  (Asked by: username_0 on 2022-01-04)
  
  Is there a way to use python and AI to compare thousands of files and detect the ones containing "unusual" content?
  Those files are supposed to have "homogeneous" configuration types with different parameters, but sometimes some of them might contain "illegal" types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.
  Here is one of too many bloc of configuration considered "correct": (the meaning doesn't matter)
  vlan XXXX
    name  XXX_NAME
  
  interface vxlan 1
    vxlan vlan XXX vni YYYY
  
  router bgp YYYY
     vlan XXXX
        rd A.B.C.D:YYYY
        route-target both YYYY:YYYY
        redistribute learned
  
  Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example
  Interface vlan AAAA
    ip address B.B.B.B/Z
  
  Because it is in only a couple of files ("rare") it is more likely that it is not authorized configuration.
  I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.
  I am wondering if AI could help with this issue?
  
  **Tags:** machine-learning, training, python
  > **Comments:**
  > **username_1**: It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?  
  > **username_2**: What makes an illegal configuration illegal? How do you detect that?  
  > **username_0**: Hey, what I call "illegal" is lines different from the most frequent ones (correct). I'll complete the question with an example.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-01-05. Score: 1
  
  Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do `interface` and `Interface` mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called "keywords" of your configurations (such as `vlan`), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.
  
  Then write rules to replace "arbitrary" data (numbers, ips, ...) with a common tag (for example , , ...). You should replace all non-keyword words (aka. tokens) with these tags. Use  for tokens you don't identify with a regex, or leave them as-is.
  
  Now we can calculate statistics of these token [n-grams](https://en.wikipedia.org/wiki/N-gram), and identify those which occur very rarely. This step will require some thingking and tuning, for example how to handle newlines (or just ignore them?), how long n-grams to use, how rare is too rare etc.
  
  This is more of a heuristic than an AI, but it works without any training data. Well, there is a human in the loop so at least it is an Intelligent solution.
  
  > **Comments:**
  > **username_0**: Nice idea @NikoNyrth. I'll give it a shot.  
  
  ---
  
  