Answers.Body: <p>Do you have a labeled dataset that specifies which connections are anomalous and which are not? You will need that to train the model.</p>
  <p>Yes, you will need all of the individual labeled data rows for training, but you could use batch inference for prediction if you only wanted to check 2-minute batches of connections at a time after the model is deployed.</p>
  <p>This is assuming that you are treating each connection as independent of the next: each connection is either anomalous or not by itself, sequencing of connections does not matter. If sequencing does matter, you'll want to consider a temporal network architecture using something like LSTM or a transformer. If this is the case, the assumption that the window of activity is 2 minutes is a strong one (why not 3 or 4 minutes)?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T07:54:52.420Z
Answers.Text: I have only normal access logs and not anomalous. The thing is that I need to identify DDoS attacks with this model. So I have to compute features per IP per 2 minutes, because only in this way you can tell if an IP tries to attack the website. So the first step is to compute the features, like how many times the IP appears in a period of 2 minutes and then train the model. If an IP appears 3 times is fine, if it appears 30 times is anomalous, for example.
Answers.UserDisplayName: Kosmylo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T07:20:09.957Z
Answers.LastActivityDate: 2022-07-09T07:20:09.957Z
Answers.UserDisplayName: dmc-au
Body: <p>I have data coming from web access logs in the following form:</p>
  <pre><code>@timestamp  ISP cache_result    client_ip   client_request_host client_request_method   client_ua   client_url  client_user content_type    ... http_response_code  major   os  os_name querystring reply_length_bytes  ts_process_time ts_timestamp    type    ua_name
  2018-04-17T08:12:32.000Z    cuaerH c rt,nlEIrnii.cec    TCP_REFRESH_MISS    25.204.184.124  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=2.2.3  25204   321 17/Apr/2018:08:12:32 -0000  testdata    Chrome
  2018-04-17T08:12:32.000Z    HeE iclirueIc rat,nrncc.    TCP_REFRESH_MISS    8.157.89.174    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-core/js/min/avada-p...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  2825    177 17/Apr/2018:08:12:32 -0000  testdata    Chrome
  2018-04-17T08:12:33.000Z    ,rrnI EnH.ceeiuclcicrat TCP_REFRESH_MISS    37.151.22.36    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  267 275 17/Apr/2018:08:12:33 -0000  testdata    Chrome
  2018-04-17T08:12:34.000Z    tn.cHer uE,lecnir aircIc    TCP_REFRESH_MISS    202.165.110.43  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  341 172 17/Apr/2018:08:12:34 -0000  testdata    Chrome
  2018-04-17T08:12:34.000Z    rneecHuraci ctInir cl.,E    TCP_REFRESH_MISS    174.201.44.32   testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-builder/assets/js/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  302 180 17/Apr/2018:08:12:34 -0000  testdata    Chrome
  </code></pre>
  <p>What I need is to come up with a model that takes as input such kind of data in batches of 2 minutes data and identify if there exists IPs that are anomalous. So, I am thinking to group the training data by IP and compute some features.</p>
  <p>But my question is - how is this possible with batches of 2 minutes, since the anomaly detection needs all the data at once to be able to train it?</p>
  
Tags: <machine-learning><algorithm-request><model-request><anomaly-detection>
Title: How is it possible to detect anomalies in batches of 2 minutes of web access logs?
UserDisplayName: Kosmylo
ThreadText: # How is it possible to detect anomalies in batches of 2 minutes of web access logs?
  
  (Asked by: username_0 on 2022-07-06)
  
  I have data coming from web access logs in the following form:
  @timestamp  ISP cache_result    client_ip   client_request_host client_request_method   client_ua   client_url  client_user content_type    ... http_response_code  major   os  os_name querystring reply_length_bytes  ts_process_time ts_timestamp    type    ua_name
  2018-04-17T08:12:32.000Z    cuaerH c rt,nlEIrnii.cec    TCP_REFRESH_MISS    25.204.184.124  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=2.2.3  25204   321 17/Apr/2018:08:12:32 -0000  testdata    Chrome
  2018-04-17T08:12:32.000Z    HeE iclirueIc rat,nrncc.    TCP_REFRESH_MISS    8.157.89.174    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-core/js/min/avada-p...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  2825    177 17/Apr/2018:08:12:32 -0000  testdata    Chrome
  2018-04-17T08:12:33.000Z    ,rrnI EnH.ceeiuclcicrat TCP_REFRESH_MISS    37.151.22.36    testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  267 275 17/Apr/2018:08:12:33 -0000  testdata    Chrome
  2018-04-17T08:12:34.000Z    tn.cHer uE,lecnir aircIc    TCP_REFRESH_MISS    202.165.110.43  testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/themes/Avada/includes/lib/assets/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  341 172 17/Apr/2018:08:12:34 -0000  testdata    Chrome
  2018-04-17T08:12:34.000Z    rneecHuraci ctInir cl.,E    TCP_REFRESH_MISS    174.201.44.32   testhost.net    GET Mozilla/5.0 (Windows NT 10.0; Win64; x64) Appl...   /wp-content/plugins/fusion-builder/assets/js/m...   -   application/javascript  ... 200 65.0    Windows 10  Windows 10  ?ver=1  302 180 17/Apr/2018:08:12:34 -0000  testdata    Chrome
  
  What I need is to come up with a model that takes as input such kind of data in batches of 2 minutes data and identify if there exists IPs that are anomalous. So, I am thinking to group the training data by IP and compute some features.
  But my question is - how is this possible with batches of 2 minutes, since the anomaly detection needs all the data at once to be able to train it?
  
  **Tags:** machine-learning, algorithm-request, model-request, anomaly-detection
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-09. Score: 1
  
  Do you have a labeled dataset that specifies which connections are anomalous and which are not? You will need that to train the model.
  
  Yes, you will need all of the individual labeled data rows for training, but you could use batch inference for prediction if you only wanted to check 2-minute batches of connections at a time after the model is deployed.
  
  This is assuming that you are treating each connection as independent of the next: each connection is either anomalous or not by itself, sequencing of connections does not matter. If sequencing does matter, you'll want to consider a temporal network architecture using something like LSTM or a transformer. If this is the case, the assumption that the window of activity is 2 minutes is a strong one (why not 3 or 4 minutes)?
  
  > **Comments:**
  > **username_0**: I have only normal access logs and not anomalous. The thing is that I need to identify DDoS attacks with this model. So I have to compute features per IP per 2 minutes, because only in this way you can tell if an IP tries to attack the website. So the first step is to compute the features, like how many times the IP appears in a period of 2 minutes and then train the model. If an IP appears 3 times is fine, if it appears 30 times is anomalous, for example.  
  
  ---
  
  