<p>Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do <code>interface</code> and <code>Interface</code> mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called &quot;keywords&quot; of your configurations (such as <code>vlan</code>), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.</p>
<p>Then write rules to replace &quot;arbitrary&quot; data (numbers, ips, ...) with a common tag (for example <code>&lt;NUMBER&gt;</code>, <code>&lt;IP&gt;</code>, ...). You should replace all non-keyword words (aka. tokens) with these tags. Use <code>&lt;?&gt;</code> for tokens you don't identify with a regex, or leave them as-is.</p>
<p>Now we can calculate statistics of these token <a href="https://en.wikipedia.org/wiki/N-gram" rel="nofollow noreferrer">n-grams</a>, and identify those which occur very rarely. This step will require some thingking and tuning, for example how to handle newlines (or just ignore them?), how long n-grams to use, how rare is too rare etc.</p>
<p>This is more of a heuristic than an AI, but it works without any training data. Well, there is a human in the loop so at least it is an Intelligent solution.</p>

CC BY-SA 4.0
2022-01-07T20:33:39.137Z
Nice idea @NikoNyrth.
I'll give it a shot.
AJN
CC BY-SA 4.0
2022-01-05T21:06:41.967Z
2022-01-07T20:40:27.983Z
2022-01-07T20:40:27.983Z
NikoNyrh
<p>Is there a way to use python and AI to compare thousands of files and detect the ones containing &quot;unusual&quot; content?</p>
<p>Those files are supposed to have &quot;homogeneous&quot; configuration types with different parameters, but sometimes some of them might contain &quot;illegal&quot; types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.</p>
<p>Here is one of too many bloc of configuration considered &quot;correct&quot;: (the meaning doesn't matter)</p>
<pre><code>vlan XXXX
  name  XXX_NAME

interface vxlan 1
  vxlan vlan XXX vni YYYY

router bgp YYYY
   vlan XXXX
      rd A.B.C.D:YYYY
      route-target both YYYY:YYYY
      redistribute learned
</code></pre>
<p>Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example</p>
<pre><code>Interface vlan AAAA
  ip address B.B.B.B/Z
</code></pre>
<p>Because it is in only a couple of files (&quot;rare&quot;) it is more likely that it is not authorized configuration.</p>
<p>I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.</p>
<p>I am wondering if AI could help with this issue?</p>

CC BY-SA 4.0
2022-01-04T15:59:01.673Z
It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?
nbro
CC BY-SA 4.0
2022-01-05T07:03:40.413Z
What makes an illegal configuration illegal? How do you detect that?
AlDante
CC BY-SA 4.0
2022-01-05T13:08:20.533Z
Hey, what I call "illegal" is lines different from the most frequent ones (correct).
I'll complete the question with an example.
AJN
<machine-learning><training><python>
Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?
AJN
# Is there a way to use AI to compare thousands of files and detect the ones containing "unusual" content?

(Asked by: username_0 on 2022-01-04)

Is there a way to use python and AI to compare thousands of files and detect the ones containing "unusual" content?
Those files are supposed to have "homogeneous" configuration types with different parameters, but sometimes some of them might contain "illegal" types of configurations. The issue is that there are too many ways to do homogenous and illegal configurations, so it is not possible to statically define them.
Here is one of too many bloc of configuration considered "correct": (the meaning doesn't matter)
vlan XXXX
  name  XXX_NAME

interface vxlan 1
  vxlan vlan XXX vni YYYY

router bgp YYYY
   vlan XXXX
      rd A.B.C.D:YYYY
      route-target both YYYY:YYYY
      redistribute learned

Thousands of files will contains such bloc of configuration (with different parameters, xxxx, yyyy...) But some might contain something else, example
Interface vlan AAAA
  ip address B.B.B.B/Z

Because it is in only a couple of files ("rare") it is more likely that it is not authorized configuration.
I could use simple regular expression to match what is considered legal or illegal, but there is too many of such blocs.
I am wondering if AI could help with this issue?

**Tags:** machine-learning, training, python

> **Comments:**
> **username_1**: It may be a good idea to provide an example of a simple "correct configuration file" and an "usual/incorrect configuration file" to have a better idea of the problem. Do you also have many examples of these configuration files, or would you need to create them artificially or collect them manually?  
> **username_2**: What makes an illegal configuration illegal? How do you detect that?  
> **username_0**: Hey, what I call "illegal" is lines different from the most frequent ones (correct). I'll complete the question with an example.  

---

## 1 answers

### Answer by username_3 on 2022-01-05. Score: 1

Here is a quick idea: first calculate the count of how many times each word occurs in these documents (I don't know whether to lowercase them or not, do `interface` and `Interface` mean different things?), and sort them in the descending order of occurrence. Most frequent words can be called "keywords" of your configurations (such as `vlan`), or maybe frequently occurring ip addresses. It is up to you whether to include ips to this list or not.

Then write rules to replace "arbitrary" data (numbers, ips, ...) with a common tag (for example , , ...). You should replace all non-keyword words (aka. tokens) with these tags. Use  for tokens you don't identify with a regex, or leave them as-is.

Now we can calculate statistics of these token [n-grams](https://en.wikipedia.org/wiki/N-gram), and identify those which occur very rarely. This step will require some thingking and tuning, for example how to handle newlines (or just ignore them?), how long n-grams to use, how rare is too rare etc.

This is more of a heuristic than an AI, but it works without any training data. Well, there is a human in the loop so at least it is an Intelligent solution.

> **Comments:**
> **username_0**: Nice idea @NikoNyrth. I'll give it a shot.  

---

