## Token Interdependency Parsing (Tipping) - Fast and Accurate Log Parsing

Shayan Hashemi, M3S, University of Oulu, Finland a

Mika M¨ antyl¨, Department of Computer Science, University of Helsinki, Finland &amp; M3S, Oulu, Finland

✦

Abstract -In the last decade, an impressive increase in software adaptions has led to a surge in log data production, making manual log analysis impractical and establishing the necessity for automated methods. Conversely, most automated analysis tools include a component designed to separate log templates from their parameters, commonly referred to as a 'log parser'.

This paper aims to introduce a new fast and accurate log parser, named 'Tipping'. Tipping combines rule-based tokenizers, interdependency token graphs, strongly connected components, and various techniques to ensure rapid, scalable, and precise log parsing. Furthermore, Tipping is parallelized and capable of running on multiple processing cores with close to linear efficiency. We evaluated Tipping against other state-of-the-art log parsers in terms of accuracy, performance, and the downstream task of anomaly detection.

Accordingly, we found that Tipping outperformed existing methods in accuracy and performance in our evaluations. More in-depth, Tipping can parse 11 million lines of logs in less than 20 seconds on a laptop machine. Furthermore, we re-implemented a parallelized version of the past IpLom algorithm to demonstrate the effect of parallel processing, and it became the second-fastest parser.

As logs keep growing in volume and complexity, the software engineering community needs to ensure automated log analysis tools keep up with the demand, being capable of efficiently handling massive volumes of logs with high accuracy. Tipping's robustness, versatility, efficiency, and scalability make it a viable tool for the modern automated log analysis task.

Index Terms -Log Parser, Strongly Connected Components, Automated Log Analysis, Large Scale Log Parsing, Parallel Processing

## 1 INTRODUCTION

I N recent years, breakthroughs in software development have led to an exponential increase in log generation, as complex systems and applications continuously produce vast amounts of log data. These logs provide insights into system performance, user behaviors, and potential security threats. However, the sheer volume and speed at which this data is produced have made it increasingly challenging for human analysts to manually process and interpret these logs effectively. As software systems grow more sophisticated and interconnected, the deluge of data, including transaction details, system statuses, and operational anomalies, continues to expand.

The log parser is one of the most crucial components in any log analysis pipeline. This tool is the foundational mechanism that interprets raw log messages into a structured format, making it crusial for further analysis. Log parsing, sometimes referred to as log clustering, is a method aimed at simplifying the complexity of free-form log messages. This process involves differentiating between a log message's constant and variable parts; the constant part (template) represents the event description, while the variable part (parameters) includes dynamic data specific to each event. As an illustration, in the log message 'Socket connected successfully to 10.10.1.10', the phrase 'Socket connected successfully to' is the template, which indicates an event. At the same time, '10.10.1.10' represents the replaceable parameter that may vary with each specific event occurrence.

Furthermore, log parsing plays a vital role in enhancing software management and security across various fields [1]. It is essential for anomaly detection [2]-[6], aiding in the identification and prediction of unusual behaviors through advanced algorithms. For debugging [7], [8], it enables developers to trace errors back to specific lines of code or external factors. Log parsing is used to identify bottlenecks, improving application efficiency and speed in performance analysis [9], [10]. Security analysis [11], [12] leverages log parsing to detect potential threats by analyzing patterns of suspicious activity. Real-time software monitoring [13] utilizes parsed logs to gain insights into user behavior and system performance, facilitating issue detection. Root cause analysis [14], [15] uses log parsing to model common event patterns, helping to identify the origins of software issues. Log parsing supports audit trails [16] by tracking data access and modifications, ensuring compliance and security. Each of these applications underscores the versatility and indispensability of log parsing in modern software engineering. The pivotal importance of log parsing has prompted the creation of numerous tools aimed at optimizing this process [17]-[24]

In log parsing, performance and accuracy stand as the twin pillars essential to its effectiveness. Performance refers to the speed at which log data can be processed and analyzed, a critical factor when dealing with large volumes of logs. Accuracy, however, involves the precision with which the log parser can identify and separate the template and parameter parts from each other, which is crucial for meaningful interpretation and making the most of down-the-stream components. However, these two often find themselves at odds. Enhancing accuracy typically involves more complex processing techniques that are computationally intensive and slower. This inverse relationship poses a significant

challenge: striving for higher accuracy can compromise the speed.

Parallel processing refers to distributing tasks across multiple processors or cores to execute operations simultaneously, enhancing speed as a result. In the context of log parsing, this approach enables the handling of extensive datasets at a faster rate while allowing for the incorporation of complex parsing algorithms that ensure high accuracy. Log parsing performance can be improved by leveraging parallel processing while maintaining accuracy as complicated methods are broken down into chunks that can be processed concurrently.

However, implementing parallel processing comes with its own set of challenges. Managing concurrent operations across different processors adds complexity to system design, necessitating mechanisms for coordinating tasks and handling potential conflicts, such as mutual exclusion. Additionally, ensuring data consistency and synchronization across multiple processing streams can be difficult, posing a risk of errors and inconsistencies in the parsed results. Thus, while parallel processing offers substantial benefits in terms of performance and capability for sophisticated methods, it requires meticulous calculations and robust architecture design to overcome these challenges.

We offer a novel parsing algorithm named 'Tipping,' which employs a rule-based tokenizer, an interdependency token graph, strongly connected components, and the mapreduce parallelism paradigm to optimize performance and efficiently parse logs. Such a structure allows for rapid execution times that scale effectively with the number of CPU cores utilized. Additionally, Tipping is designed with various options, providing extensive flexibility for finetuning according to specific needs. This ensures that our algorithm is adaptable to a wide range of environments and requirements.

Finally, we demonstrate that the Tipping algorithm outperforms state-of-the-art methods across four benchmarks, highlighting its accuracy and performance. Tipping achieves higher accuracy and outputs fast parsing times in the LogPai and LogPM tests for accuracy and performance and processes a massive volume of logs faster than other methods in the LogLead test for computational efficiency. Additionally, Tipping excels in LogLead's anomaly detection downstream task as well. These results underscore Tipping's robustness and adaptability, making it a performance, accurate, and versatile tool for various log analysis applications.

## 2 RELATED WORKS

In a brilliant study, Drain [18] is designed to process logs in a streaming and timely manner. Unlike traditional log parsers focusing on offline batch processing and struggling with the rapidly increasing volume of logs, Drain introduced a fixed depth parse tree to accelerate the parsing process. This parse tree, crucial for guiding the log group search process, encodes specially designed parsing rules within its nodes. Its fixed depth tree structure and specially designed parsing rules significantly enhance the speed and accuracy of log parsing, making it an excellent tool for online log parsing.

Another study introduces the Brain [17], an innovative log parsing approach that leverages a bidirectional parallel tree to transform semi-structured logs into a structured format, aimed at overcoming the limitations of existing parsers in software systems. By identifying the longest common pattern across logs as a likely component of the log template, Brain efficiently groups logs and hierarchically integrates constant words into these patterns to form complete templates. This method significantly enhances parsing accuracy and efficiency across various software logs. Furthermore, Brain's process encompasses preprocessing for word splitting and variable filtering, initial group creation based on common patterns, and using a bidirectional tree to systematically add constant words, culminating in the generation of structured log templates. Brain offers improved stability and processing speed for large volumes of log data.

NuLog, introduced in [25], is a self-supervised log parsing technique leveraging transformer architecture to address the challenge of parsing massive volumes of semi-structured logs generated by large-scale software systems. Unlike traditional methods that rely on heuristics or manual rule extraction, NuLog uses masked language modeling (MLM) to predict the presence of words in log messages based on their context, distinguishing between constant and variable parts without requiring domain knowledge. The methodology encompasses tokenization, masking, and a multi-head self-attention mechanism with positional encoding and a feed-forward network, outputting log templates alongside numerical vector summarizations. This enables high parsing accuracy and supports downstream anomaly detection tasks in both supervised and unsupervised scenarios. NuLog's approach significantly advances automated log analysis, offering broad applicability and minimal human intervention.

Another ingenious paper presents Spell [19], a novel streaming log parser that leverages the Longest Common Subsequence (LCS) method for parsing system event logs in real time into structured data. Moreover, Spell dynamically extracts log patterns and updates its log structure by comparing incoming log entries against stored LCS sequences within an LCSMap data structure. Furthermore, It utilizes a pre-filtering step using a prefix tree and a simple loop approach to efficiently identify existing message types, significantly reducing the computational load by applying LCS computation selectively. Spell is considered an effective solution for online log monitoring and analysis, offering immediate insights into system behavior and enhancing log management capabilities.

MoLFI, introduced in [20], proposes an innovative approach by framing it as a multi-objective optimization problem, efficiently solved using the NSGA-II [26] algorithm to balance the frequency and specificity of templates. This method requires no domain knowledge to address the challenges of rapidly evolving log formats. MoLFI produces a Pareto optimal set of templates that adeptly navigates the trade-offs between template commonality and uniqueness through multiple steps of preprocessing, solution encoding, and iterative refinement using genetic operations. Overall, MoLFI introduced a completely different approach compared to other well-known log parsers at the time.

Logram [21] is another log parsing method that employs n-gram dictionaries to address the log parsing problem efficiently. By leveraging the intuition that frequently occurring n-grams likely represent static text and rare n-grams

indicate dynamic variables, Logram generates n-gram dictionaries from log messages and uses these dictionaries to parse and distinguish between static and dynamic components of logs. This process involves preprocessing logs to extract tokens, creating dictionaries for efficient queries, and employing an automated method to determine the occurrence threshold for dynamic variable identification. In their own evaluations, Logram showed improved parsing accuracy and efficiency.

ULP (Unified Log Parser) [22] is a log parsing tool that enhances accuracy and efficiency through a novel combination of string matching and local frequency analysis. ULP effectively differentiates between static and dynamic components of log messages by grouping log events based on string similarity and then applying frequency analysis within these groups. This method outperformed its rival competition at the time. The parser involves preprocessing to identify trivial dynamic tokens, grouping similar events for targeted analysis, and generating templates through local frequency insights, showcasing a significant improvement in parsing large log files and offering a solution for complex log analysis tasks.

The log parser LenMa (Length Matters) [27] introduces a unique clustering approach, leveraging the length of words within each message to distinguish between fixed and variable components, thereby facilitating efficient online template generation. LenMa simplifies the process by employing word length vectors to compute similarity scores between new log entries and existing clusters using cosine similarity. This innovative approach enables real-time, dynamic clustering of log messages, efficiently adapting to the continuous influx of logs. LenMa proves to be an effective and straightforward method for managing vast volumes of log data in a scalable solution.

IpLom (Iterative Partitioning Log Mining) [28], [29] is a multi-step log clustering approach that hierarchically partitions log messages into different groups. The first step of partitioning divides log messages into groups based on the number of tokens. The second step further processes each of these groups by splitting them based on the token position with the least variability, i.e., they are the most likely to be templates. The third step searches for a bijective relationship with token positions that co-vary. Although presented already 15 years ago, IpLom has shown promising accuracy and speed in independent tests [1]. The main drawback of IpLom is that varying numbers of tokens in log parameters can cause incorrect groups, and the algorithm does not include the ability to correct these groups.

AEL (Abstracting Execution Logs) [30] is based on four hierarchical steps: anonymize, tokenize, categorize, and reconcile. The anonymized step uses external heuristics to convert log message parameters to generic tokens. The tokenize step uses the number of tokens and parameters from the previous step to group log messages. The categorization step goes through existing groups and performs token-bytoken comparison and clustering. Finally, the reconcile step tries to account for the lack of heuristics of the anonymized step, and merges log clusters if too many execution events have been created. AEL has also shown good accuracy and speed in independent tests [1]. The main drawback of the algorithm is the need to have A-priori heuristics on how to recognize log message parameters.

## 3 PROPOSED METHOD

This section outlines the structure of our methodology, breaking it down into its components. Our algorithm is structured into two phases, unlike other methods that perform parsing in a single phase. During the initial phase (Sec. 3.1 to 3.3), it gathers information from log messages. In the subsequent phase (Sec. 3.4 to 3.10), the collected information is employed for log parsing. The two-phase approach enhances the algorithm's accuracy and allows for the utilization of parallel processing.

Figure 3 shows an overview of our approach, while the following subsections explain each phase in detail.

## 3.1 Tokenizer

The first component of our parser is the tokenizer, which is common in almost every NLP task. The tokenizer breaks down log messages into smaller substrings that make the data more suitable for algorithms to process. Unlike most previous methods that split text at white space locations, our approach to tokenization allows for domain knowledge to be passed from users.

Our tokenizer requires three inputs for the tokenization process: white special patterns, black special patterns, and symbols. While the special tokens are optional, the symbols set has the default value of { ( ) [ ] { } = , ∗} .

The white special patterns refer to an optional list of regular expressions that the algorithm should recognize as templates during the parsing process. These patterns are earmarked to be excluded from being recognized as parameters, ensuring they remain intact within the entire parsing process and will become a part of the output templates.

Conversely, black special patterns are designated to be recognized as parameters unequivocally, ensuring their segregation from the template content. White and black special patterns allow for more nuances in the parsing process and results, resolving problems such as maintaining templateshared tokens [1] and execution environment bias parameters [1].

Lastly, the symbols input consists of a curated set of punctuation characters. These symbols, alongside whitespace characters, serve as delimiters in the tokenization process to define the boundaries between non-special tokens. Our algorithm achieves a sophisticated and tailored approach toward log message tokenization by leveraging these three inputs.

The tokenizer initiates its process by scanning the log message to identify all occurrences of white special patterns, which are then marked as white tokens. Subsequently, the algorithm examines the remaining portions of the log message to locate black special patterns and flag them as black tokens. Following the identification of white and black tokens, the algorithm continues with an iteration over the residual segments of the log message while employing symbols and white space characters as delimiters to fragment the message. Uniquely, while splitting the message, the algorithm retains these delimiter characters as separate tokens.

Fig. 1. An overall view of Tipping's components, internal workflow, relation, and execution order.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00645v1-figure-fac2e0ff-a5f0-467d-8bbf-873fabfafb2c.jpg)

After identifying white and black tokens, the remainder of the log message is fragmented by symbols and whitespaces. The tokenizer concludes its process by compiling the sequence of tokens and returning it as the tokenized message.

## 3.2 Token Filter

The token filter receives tokenized messages and filters out unwanted tokens based on user-defined parameters. This allows for better leverage of domain knowledge while being capable of producing accurate results using the default values. By default, the token filter only keeps the white special tokens and purely alphabetic tokens.

## 3.3 Occurences and Co-occurrences

For further down-the-line components, keeping track of both token occurrences and their co-occurrences is essential.

Token occurrence refers to the number of times that a particular token appears within a given dataset. On the other hand, co-occurrence involves tracking instances where two tokens appear together within the same message. Both occurrence and co-occurrence could be stored in a single sparse matrix data structure. However, storing occurrences and co-occurrences in a hash map data structure is more suitable, as row-or-column-wise access is not required. Within the hash map data structure, each key is a singleor-two-members set of token(s), indicating the occurrences and co-occurrences, while the value is the count for that set.

## 3.4 Interdependency Graph

Before going into the interdependency graph, it's essential to define dependency in the context of this paper. Token dependency, inspired by recommender systems' support concept [31] and conditional probability, is defined as the

ratio of log messages in which both tokens appear together to the number of log messages that contain the dependent one. More in-depth, the dependency of the token A to B is the co-occurrence(s) of A and B together divided by the total occurrences of A . To formalize this, if S A represents the set of messages containing token A while S B denotes those with token B , the dependency of token A on token B can be mathematically expressed based on these occurrences as follows:

$$d e p ( A, B ) = \frac { | S _ { A } \cap S _ { B } | } { | S _ { A } | }$$

On the other hand, the interdependency graph is a directed graph constructed for each log message individually, where nodes (vertices) are tokens and edges (arcs) are the dependence of the source nodes' tokens on destination nodes' tokens.

## 3.5 Strongly Dependent Token Group

A strongly dependent token group is a subset of tokens within a message characterized by their mutual dependence on one another. Conversely, tokens outside this subset remain independent from the group, meaning either they do not depend on the tokens within the group or the group tokens do not depend on them. It is worth mentioning that there is at least one token group in a message, while the maximum number is equal to the number of unique tokens in that message.

The intuition behind grouping tokens based on their dependency is that when token A depends on token B , and A is identified as part of the template, inevitably, B is also part of the template as token A is unlikely to occur without token B . Therefore, in our context, tokens within a token group are expected to appear together.

A threshold hyper-parameter, denoted as θ , is required to compute strongly dependent token groups (reasons are going to be discussed later). The process begins by constructing a token interdependency graph from the message. Then, edges with weights below the θ are then disregarded. Next, strongly dependent token groups are identified using strongly connected component algorithms, such as the Tarjan [32] or Kosaraju-Sharir [33] algorithm.

There are two reasons for disregarding edges below the threshold. First, it helps eliminate low dependencies often caused by noise. Second, since any two tokens within a single message are mentioned together at least once, a baseline of dependency always exists among all tokens in a message, leading to a fully connected interdependency graph. On the other hand, strongly connected components algorithms focus only on the presence of edges, not their weights, which eventually results in a single strongly dependent token group.

Algorithm 1 outlines all the steps required to compute the strongly dependent token groups.

## 3.6 Anchor Tokens Selection

After identifying strongly dependent token groups, the largest one is selected as the anchor group, and its members are designated as the anchor tokens.

Algorithm 1 Anchor token identification given message m and threshold θ .

```

or Algorithm 1 Anchor token identification given
it
and threshold 0.
B
    t <- tokenize(m)
else
    t' <- filter(t)
else
    g <- interdep_graph(t')
else
    for e \in edges(g) do
        if e < \theta then
            remove(e)
        end if
    end for
    anchor <- \{\}
d
    for c \in connected_components(g) do
        if |anchor| < |c| then
            anchor \gets c
s'
        end if
    end for
    return  anchor
    


    \^ =   \^ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
```

## 3.7 Anchor-token-based Clustering

After identifying anchor tokens, messages are clustered into distinct groups according to their anchor tokens. Then, event IDs could be obtained from clusters by computing cluster hash or its index. Should the desired outcome be solely the event IDs, the algorithm concludes at this juncture. However, if there is a request for templates or parameter masks in addition to event IDs, the algorithm proceeds to produce the next three optional steps.

## 3.8 Cluster Common Tokens

Common tokens within each cluster are those present across all messages in the cluster, effectively representing the intersection of tokens in all messages. Subsequently, these common tokens are later leveraged to generate event templates and parameter masks, offering a refined approach to parsing and analysis.

## 3.9 Event Templates

Unlike previous methods, which produced a single template for each unique event, our method produces a set of templates for each event to handle the dynamicity occurring in log messages.For instance, Table 1 illuminates an example from the Proxifier dataset. In the example, all messages are representations of the same event. However, due to optional parameters within the log message structure [1], templates differ from each other.

The process of generating templates starts with a cluster of log messages and their common tokens. Initially, it involves processing these messages to create preliminary templates, where common tokens are retained, and the remaining elements are replaced with placeholders. To streamline the sequence, consecutive placeholders are merged into a single one. Subsequently, to make the template representable as a string, placeholders are replaced by a specific parameter symbol, such as ' &lt; &gt; * ', to indicate variable parts. The final step aggregates these templates into a set, producing the templates for that cluster.

TABLE 1 An example of a situation where a single event may have different template representations.

| Message 1 Template 1   | chrome.exe *64 - c.cnzz.com:80 close, 0 bytes sent, 0 bytes received, lifetime 00:10 <*> close, <*> bytes sent, <*> bytes received, lifetime <*>                                          |
|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Message 2 Template 2   | chrome.exe *64 - nos.netease.com:443 close, 329 bytes sent, 3490 bytes (3.40 KB) received, lifetime 01:00 <*> close, <*> bytes sent, <*> bytes <*> received, lifetime <*>                 |
| Message 3 Template 3   | chrome.exe *64 - css.sohu.com:80 close, 1463 bytes (1.42 KB) sent, 766 bytes received, lifetime 00:22 <*> close, <*> bytes <*> sent, <*> bytes received, lifetime <*>                     |
| Message 4 Template 4   | chrome.exe *64 - manhua.163.com:443 close, 8544 bytes (8.34 KB) sent, 102059 bytes (99.6 KB) received, lifetime 00:08 <*> close, <*> bytes <*> sent, <*> bytes <*> received, lifetime <*> |

## 3.10 Parameter Masks

After identifying the common tokens across a cluster of messages, we initialize the parameter mask with an empty string for each message. Then, the parameter mask is populated by iterating over the message tokens, appending n zeros if the token is present in the common tokens or n ones otherwise, where n is the number of characters in each token. Repeating this for all messages generates parameter masks for the entire cluster.

## 4 EXPERIMENTS

Given that our algorithm relies on parallel computing, we implemented our algorithm in Rust due to its fearless concurrency, exceptional performance, and seamless compatibility with other languages. However, since Python is the dominant language in data science and AI space, we also released a Python version of our package (the core is still written in Rust), which will be utilized in the upcoming experiment. Tipping Python package can be installed from pypi 1 , while the Rust package is available on crates.io 2 .

We performed four experiments, see Sections 4.1 to 4.4, to answer the following research questions:

- · RQ1 : How accurate is Tipping in log parsing? Experiments in Sections 4.1 and 4.2
- · RQ2 : How fast is Tipping in log parsing? Experiments in Sections 4.2 and 4.3
- · RQ3 : What is the impact of Tipping as a parser on the down-the-stream task? Experiment in Section 4.4

## 4.1 LogPai's Benchmark

## 4.1.1 Experiment depiction

LogPai's benchmark [24], published in 2019, has become a canonical accuracy evaluation benchmark for log parsers since its introduction. The benchmark comprises 2,000 manually labeled samples across 16 public open datasets. It primarily uses group accuracy and the F 1 score as its main metrics. Additionally, it includes time measurement during parsing as a performance metric. However, it is observed that performance might not be fully representative of realworld scenarios due to the limited number of samples in each dataset [1]. It's important to note that although parsers are evaluated using larger datasets in the original paper, to

- 1. https://pypi.org/project/tipping

2. https://crates.io/crates/tipping-rs

the best of our knowledge, these larger datasets have not been made available in the GitHub repository.

LogPai permits the tuning of hyper-parameters specific to each dataset. However, we allocated no more than five minutes to tune the parameters for each dataset, as Tipping can reach maximum accuracy provided with the appropriate settings. Moreover, we were cautious not to overcomplicate the tuning process, aiming to maintain as much generalizability as possible in our approach.

It's worth mentioning that since both group accuracy and F 1 score metrics are hardware-agnostic, we did not need to rerun the other state-of-the-art algorithms with our benchmark machine. Instead, we borrowed the results from the public GitHub repository 3 .

In favor of transparency, we made our implementation of Tipping in LogPai's benchmark, which is a fork of the main repository, publicly available . 4 Furthermore, we intend to merge this with the original repository after publishing the paper.

## 4.1.2 Experiment results

In Table 2, Tipping demonstrates strong performance across various datasets, achieving perfect scores (1.00) in multiple cases such as Apache, HDFS, HealthApp, Proxifier, Spark, and Windows. While it shows slightly lower performance in others, like Android, BGL, HPC, Hadoop, Linux, Mac, OpenSSH, OpenStack, Thunderbird, and Zookeeper, Tipping's scores are consistently high, mostly above 0.90, with the lowest score being 0.81 in the OpenSSH dataset. The results illustrate Tipping's robustness to a diverse range of log datasets. Its performance is competitive compared to other state-of-the-art parsers, often matching or outperforming them in almost all datasets, underscoring its effectiveness as a log parsing tool.

Table 3 showcases the F 1 scores achieved by log parsers in the LogPai's benchmark, where Tipping and Brain both scored consistently at 1.00 across almost every dataset, showcasing their top-tier accuracy. NuLog, however, showed variability in performance in the available datasets as its results were not available for all datasets. Other parsers like Spell, MoLFI, Logram, and ULP showed more variability in their scores, with Tipping outperforming them, highlighting its effectiveness as a log parsing tool compared to other state-of-the-art approaches.

3. https://github.com/logpai/logparser

4. https://github.com/M3SOulu/logparser

TABLE 2

A detailed comparison of the Tipping model's performance against other leading state-of-the-art models in LogPai's benchmark using the Group Accuracy as the evaluation metric.

| Dataset     |   Tipping |   Brain | NuLog   |   Drain |   Spell |   MoLFI |   Logram |   ULP |
|-------------|-----------|---------|---------|---------|---------|---------|----------|-------|
| Android     |      0.94 |    0.96 | 0.83    |    0.91 |    0.92 |    0.63 |     0.79 |  0.84 |
| Apache      |      1    |    1    | 1.00    |    1    |    1    |    1    |     0.31 |  1    |
| BGL         |      0.98 |    0.99 | 0.98    |    0.96 |    0.79 |    0.96 |     0.59 |  0.93 |
| HDFS        |      1    |    1    | 1.00    |    1    |    1    |    1    |     0.93 |  1    |
| HPC         |      0.99 |    0.94 | 0.95    |    0.89 |    0.65 |    0.81 |     0.91 |  0.95 |
| Hadoop      |      0.95 |    0.95 |         |    0.95 |    0.78 |    0.95 |     0.45 |  0.99 |
| HealthApp   |      1    |    1    | 0.88    |    0.78 |    0.64 |    0.32 |     0.27 |  0.9  |
| Linux       |      0.97 |    1    |         |    0.69 |    0.6  |    0.29 |     0.36 |  0.36 |
| Mac         |      0.91 |    0.94 | 0.82    |    0.79 |    0.76 |    0.62 |     0.57 |  0.81 |
| OpenSSH     |      0.81 |    1    |         |    0.79 |    0.55 |    0.54 |     0.61 |  0.43 |
| OpenStack   |      1    |    1    | 0.97    |    0.73 |    0.76 |    0.21 |     0.33 |  0.49 |
| Proxifier   |      1    |    1    |         |    0.53 |    0.53 |    0    |     0.5  |  0.02 |
| Spark       |      1    |    1    | 1.00    |    0.92 |    0.9  |    0.42 |     0.28 |  0.92 |
| Thunderbird |      0.98 |    0.97 |         |    0.96 |    0.84 |    0.64 |     0.55 |  0.68 |
| Windows     |      1    |    1    | 0.99    |    1    |    0.99 |    0.41 |     0.69 |  0.41 |
| Zookeeper   |      0.99 |    0.99 |         |    0.97 |    0.96 |    0.84 |     0.72 |  0.99 |

TABLE 3

A detailed comparison of the Tipping model's performance against other leading state-of-the-art models in LogPAI's benchmark using F 1 score as the evaluation metric.

| Dataset     |   Tipping |   Brain | NuLog   |   Drain |   Spell |   MoLFI |   Logram |   ULP |
|-------------|-----------|---------|---------|---------|---------|---------|----------|-------|
| Android     |      1    |       1 | 0.97    |    1    |    0.99 |    0.85 |     0.98 |  0.97 |
| Apache      |      1    |       1 | 1.00    |    1    |    1    |    1    |     0.64 |  1    |
| BGL         |      1    |       1 | 1.00    |    1    |    0.96 |    1    |     0.96 |  1    |
| HDFS        |      1    |       1 | 1.00    |    1    |    1    |    1    |     0.99 |  1    |
| HPC         |      1    |       1 | 0.99    |    0.99 |    0.99 |    0.98 |     0.99 |  0.99 |
| Hadoop      |      1    |       1 |         |    1    |    0.92 |    1    |     0.78 |  1    |
| HealthApp   |      1    |       1 | 1.00    |    0.92 |    0.89 |    0.78 |     0.59 |  0.99 |
| Linux       |      1    |       1 |         |    0.99 |    0.94 |    0.72 |     0.54 |  0.48 |
| Mac         |      0.99 |       1 | 0.75    |    0.98 |    0.96 |    0.93 |     0.89 |  0.98 |
| OpenSSH     |      1    |       1 |         |    1    |    0.92 |    1    |     0.98 |  0.94 |
| OpenStack   |      1    |       1 | 1.00    |    0.99 |    0.99 |    0.73 |     0.74 |  0.83 |
| Proxifier   |      1    |       1 |         |    0.78 |    0.83 |    0.74 |     0.75 |  0.74 |
| Spark       |      1    |       1 | 1.00    |    0.99 |    0.99 |    0.51 |     0.48 |  0.99 |
| Thunderbird |      1    |       1 |         |    1    |    0.99 |    1    |     0.99 |  1    |
| Windows     |      1    |       1 | 1.00    |    1    |    1    |    0.91 |     0.91 |  0.99 |
| Zookeeper   |      1    |       1 |         |    1    |    1    |    1    |     0.92 |  1    |

## 4.2 LogPM Benchmark

fering significantly larger datasets than those found in LogPai.

## 4.2.1 Experiment depiction

While LogPai has gained widespread popularity within the field of log parsing as a benchmark for evaluating the accuracy of log parsing methods, it lacks significant features. The benchmark's reliance on group accuracy as the defining metric and containment of insufficient quantity of samples to accurately reflect real-world performance scenarios have prompted researchers to seek alternatives. In response to these limitations, the recently published LogPM benchmark [1] has been introduced, addressing some of the criticisms of LogPai. LogPM distinguishes itself by emphasizing the detection of parameters at the character level. More indepth, drawing inspiration from the field of computer vision, specifically semantic segmentation, LogPM introduces a new output for log parsers called the 'Parameter Mask.' This novel approach is accompanied by a new metric, the Parameter Mask Agreement (PMA), designed to compare the produced parsers' parameter masks against the ground truth. Furthermore, LogPM endeavors to better replicate real-world prerequisites for performance evaluation by of-

In favor of transparency, we merged our implementation of Tipping in LogPM benchmark into the main publicly available repository . 5

## 4.2.2 Experiment results

In Table 4, Tipping demonstrates a consistently higher parsing accuracy performance across a variety of datasets compared to its counterparts: Brain, Drain, Spell, and Lenma, as indicated by its average accuracy of 0.71. While Brain and Lenma show strong performances in specific datasets, they lack the overall consistency of Tipping. Particularly, Brain's strengths in certain areas like HPC and Hadoop are counterbalanced by weaker performances in others, and Lenma, despite its high marks in HPC and Zookeeper, does not maintain this level of accuracy across all datasets. On the other hand, Drain and Spell exhibit lower overall accuracies. For instance, Drain excels in the SSH dataset but falls short in broader applications, indicating a niche area of strength. Spell's performance is even less consistent, further

5. https://github.com/M3SOulu/LogPMBenchmark

TABLE 4

An in-depth comparison of the Tipping's performance with other leading state-of-the-art parsers in the LogPM benchmark , utilizing Parsing Accuracy as the evaluation metric.

| Dataset   |   Tipping |   Brain |   Drain |   Spell |   Lenma |
|-----------|-----------|---------|---------|---------|---------|
| HPC       |      0.47 |    0.84 |    0.79 |    0.6  |    0.93 |
| Zookeeper |      1    |    0.91 |    0.99 |    0.92 |    1    |
| Android   |      0.8  |    0.6  |    0.04 |    0.31 |    0.37 |
| Apache    |      0.49 |    0.19 |    0.19 |    0.19 |    0.7  |
| Hadoop    |      0.74 |    0.86 |    0.72 |    0.84 |    0.94 |
| HDFS      |      1    |    1    |    0.72 |    0.72 |    0.83 |
| Linux     |      0.92 |    0.9  |    0.87 |    0.87 |    0.92 |
| Openstack |      0.72 |    0.24 |    0.12 |    0.06 |    0.76 |
| Proxifier |      0.9  |    0.13 |    0    |    0.07 |    0    |
| SSH       |      0.04 |    0.69 |    0.91 |    0.23 |    0.04 |
| Avg.      |      0.71 |    0.63 |    0.53 |    0.48 |    0.65 |

emphasizing Tipping's advantage in delivering higher accuracy across diverse datasets. Overall, Tipping's performance underscores its reliability and capacity to adapt to varied data types, making it a preferred choice for comprehensive LogPm parsing tasks where data variability is a significant factor.

TABLE 5 An in-depth comparison of the Tipping's performance with other leading state-of-the-art parsers in the LogPM benchmark , utilizing Parameter Mask Agreement as the evaluation metric.

| Dataset   |   Tipping |   Brain |   Drain |   Spell |   Lenma |
|-----------|-----------|---------|---------|---------|---------|
| HPC       |      0.88 |    0.8  |    0.78 |    0.77 |    0.82 |
| Zookeeper |      0.97 |    0.96 |    0.97 |    0.95 |    0.97 |
| Android   |      0.65 |    0.42 |    0.18 |    0.35 |    0.3  |
| Apache    |      0.85 |    0.5  |    0.5  |    0.36 |    0.77 |
| Hadoop    |      0.81 |    0.89 |    0.82 |    0.89 |    0.9  |
| HDFS      |      1    |    0.89 |    0.78 |    0.84 |    0.85 |
| Linux     |      0.96 |    0.86 |    0.86 |    0.86 |    0.86 |
| Openstack |      0.92 |    0.91 |    0.74 |    0.7  |    0.92 |
| Proxifier |      0.95 |    0.79 |    0.62 |    0.62 |    0.48 |
| SSH       |      0.66 |    0.95 |    0.98 |    0.79 |    0.89 |
| Avg.      |      0.87 |    0.8  |    0.72 |    0.71 |    0.8  |

In Table 5, presenting LogPM's parameter mask agreement scores, Tipping emerges as a leading method with an average agreement score of 0.87, showcasing its superior performance across various datasets. This performance is indicative of Tipping's effectiveness in accurately identifying and matching parameter masks, a critical aspect of modern log parsing [34]. Comparatively, Brain and Lenma each hold an average agreement score of 0.80, indicating a commendable performance but still trailing behind Tipping. Both methods show strengths in certain areas, such as Lenma's impressive accuracy in Hadoop and Brain's exceptional performance in SSH, illustrating their potential in specific contexts. However, Tipping's across-the-board excellence demonstrates a more universally applicable solution to parameter mask agreement challenges. Drain and Spell, with average scores of 0.72 and 0.71, respectively, lag further behind, particularly struggling in datasets like Android and Apache, where Tipping excels. Overall, Tipping's performance is one level above its competition.

In Table 6, Tipping demonstrates remarkable efficiency, with an average speed score that is the fastest performance

TABLE 6

An in-depth comparison of the Tipping's performance with other state-of-the-art parsers in the LogPM benchmark's speed test . The numbers are normalized for each dataset by dividing them by the lowest value, making the comparison between different datasets size-agnostic.

| Dataset   |   Tipping |   Brain |   Drain |   Spell |   Lenma |
|-----------|-----------|---------|---------|---------|---------|
| HPC       |      2.25 |    3    |    1    |   21.45 |   22.38 |
| Zookeeper |      2.17 |    4.37 |    1    |   13.16 |   30.98 |
| Android   |      1    |    1.47 |    7.26 |   16.68 | 4435.41 |
| Apache    |      2.98 |    6.69 |    1    |   13.68 |  126.45 |
| Hadoop    |      1    |    1.22 |    3.52 |   21.06 |    9.03 |
| HDFS      |      1    |    2.15 |   29.85 |   22.61 |   11.42 |
| Linux     |      1.8  |    1.67 |    1    |   23.03 |   22.59 |
| Openstack |      2.79 |    3.74 |    1    |   43.43 |   44.24 |
| Proxifier |      2.58 |    3.35 |    1    |    6.75 |   23.67 |
| SSH       |      2.83 |    6.88 |    1    |   17.77 |   42.89 |
| Avg.      |      1    |    1.69 |    2.33 |    9.78 |  233.73 |

among the parsers, illustrating its exceptional speed in processing log data across various datasets. This performance is especially noteworthy in datasets like Android, Hadoop, and HDFS, where Tipping matches or outpaces its competitors by a significant margin. Meanwhile, Brain and Drain offer competitive speeds in certain datasets, but they do not consistently match Tipping's rapid processing times across the board. Spell and Lenma, on the other hand, show significantly slower parsing speeds in comparison, with Lenma's performance in the Android dataset highlighting a stark difference in efficiency. Lenma's average speed score is drastically higher due to its outlier performance in handling the Android dataset, indicating a potential inefficiency in processing log data with a large number of templates. Tipping's uniform speed across diverse datasets underscores its efficiency and reliability, making it capable of swiftly handling large volumes of log data and ideal for various environments.

## 4.3 Computational Efficiency with LogLead

## 4.3.1 Experiment depiction

LogLead 6 is a tool designed to benchmark log representations and anomaly detection algorithms, such as log parsing results [35]. It features integrations with many public datasets and log parsers. LogLead is built with efficiency in mind, achieved by using the Polars 7 library, a notably faster alternative to Pandas . As LogLead integrates existing 8 datasets and parsers, it offers an environment to evaluate the computational efficiency of different log parsers over different datasets. Unlike the experiments in the previous sections, no dataset-specific or parser-specific optimizations were made, and as such, are not available on the LogLead platform.

The scripts we used for our LogLead experiments are available . 9 For the purposes of this experiment, we first loaded and preprocessed log data from nine different systems from past studies. This preprocessing involved loading

- 6. https://github.com/EvoTestOps/LogLead

7. https://github.com/pola-rs/polars

- 8. https://github.com/pandas-dev/pandas
- 9. https://github.com/EvoTestOps/LogLead/tree/main/demo/ parser benchmark

data into Polars data frames and normalizing it using a standard regular expression in the LogLead Enhancer class. The datasets included BGL, Liberty, Spirit, and Thunderbird, originating from real-world high-performance computing usage [36]. The Hadoop [37] and HDFS [38] datasets contain logs from experiments conducted on distributed file systems. Additionally, Trainticket [39] and Web-shop 10 datasets represent experiment logs from modern microservice-based systems [40]. Finally, the Android2 dataset, not to be confused with the Android dataset, originates from long-lasting Android reliability testing runs [41].

After preparing the logs, we parsed them with seven log parsers and measured the time. To assess performance differences, we tested all datasets with shares ranging from 1/1024 to 1/1, with each step doubling the amount of data parsed. This arrangement allows us to easily infer whether the parsing time of a particular parser grows linearly, as we would expect the parsing time to double at each step. We conducted the measurements on a local machine (LM) with an Intel® Core™ i7-1365U Processor (two performance cores (max 5.2 GHz) and eight efficiency cores max(3.9 GHz)) with 32Gb of physical memory, and a cloud virtual machine VM with Intel Core Processor (Broadwell, IBRS) CPU 2,4Ghz, 28 cores, and 224GB memory. It should be noted that the LM has more powerful processor cores, while the VM has a higher number of cores available.

Parser implementations for AEL, Brain, and IpLom were taken from [24]. For Spell 11 and LenMa 12 , we used the implementations available in the footnotes. For Drain, we used the so-called Drain3 13 implementation. In the Drain configuration file, we disabled Drain's internal log message normalization via regular expressions and disabled profiling to ensure maximal parsing speed. For other parsers, no such settings were available.

As our purpose was to investigate the benefits of parallelization, we decided to implement a paralyzation version of the IpLom parser. We named it PlIpLom, Pl being short for Polars. We followed the original IpLom paper and the Python-based source code implementation from [24], but whenever we encountered a loop, we replaced it with native Polars operations if possible. Native Polars operations are known for their efficiency.

## 4.3.2 Experiment results

During our experiments, we set a run-time limit of 10 minutes (600 seconds for each parser on each dataset). Accordingly, if a run exceeded 10 minutes, we allowed it to finish, but no further runs were executed. The remaining results are marked as 'Out of Time Limit' (OTL) in the tables from 7 to 21. We also mark crashed runs as 'Did Not Finish' (DNF). We conducted a re-run for all crashes; however, if the re-run also failed to complete, we marked it as DNF.

We have two small log datasets. The results for the Hadoop dataset (177,592 log lines) on the local machine (LM) and virtual machine (VM) are shown in Tables 7 and

10. https://github.com/GoogleCloudPlatform/microservices-demo

11. https://github.com/bave/pyspell

12. https://github.com/keiichishima/templateminer

13. https://github.com/logpai/Drain3/

8, respectively. Tipping, Drain, and PlIplom are the three fastest parsers in both the VM and LM settings. Similar to Hadoop, TrainTicket is a small dataset (272,270 log lines), and its results are documented in Tables 9 and 10. In the TrainTicket dataset, Drain is the fastest on the LM, followed by Tipping and Iplom. However, the performance ranking on the VM changes, with Tipping being the fastest, followed by Drain and Iplom.

Next, we have four medium-sized datasets.

Web-shop, with 3,958,203 log lines, presents its results for the local machine (LM) and virtual machine (VM) in Tables 11 and 12, respectively. In the LM setting, the top three parsers are PlIplom, Tipping, and Drain, whereas in the VM, the order is Tipping, PlIplom, and Drain. Here, we begin to observe meaningful differences in parsing speed, with the difference between the first and third parsers approximately 9x in LM and 8x in VM.

The results for the BGL dataset, containing 4,747,963 log lines, are displayed in Tables 13 and 14. In both computing environments, the top three parsers, in order, are Tipping, PlIplom, and Drain.

The Android2 dataset, with 5,203,599 log lines, shows results in Tables 15 and 16. Tipping is the fastest parser in both the VM and LM, while the second and third places alternate between Drain and PlIplom in the VM and LM, respectively.

Finally, the HDFS dataset, which consists of 11,175,629 log lines, is presented in Tables 17 and 18. In the LM, PlIplom is the fastest, followed by Tipping and Drain, while in the VM, Tipping leads, followed by PlIplom and Drain.

Finally, we have three large datasets executed only in the virtual machine (VM), as the local machine's memory size was not sufficient. In the VM, none of the parsing executions resulted in fully parsed datasets, as all of the parsers either exceeded the time limit or ran out of memory.

The Thunderbird dataset, with 211,212,192 log lines, shows parsing results in Table 19. The fastest parsers, in order, were Tipping, Iplom, and Drain. This order is consistent across the remaining two large datasets. The Liberty and Spirit datasets' results are shown in Tables 21 and 20, respectively. Compared to its competitors, the only drawback noted for Tipping is its inability to parse half of the Liberty data, a task that PlIplom manages to accomplish.

## 4.3.3 Summary

To summarize, we conducted speed test experiments on two computing platforms with nine datasets, totaling 15 experiments, as we were unable to run the three largest datasets on our local machine. We found that Tipping was the fastest parser in 12 out of the 15 experiments. In two experiments on the local machine (HDFS and WebShop), Tipping was second to PlIplom but then performed faster on the virtual machine, which has more CPU cores, though the cores themselves are slower. A similar result was observed in the local machine with TrainTicket, where Drain was the fastest. This provides evidence of Tipping's superior parallelization capability. The second fastest parser was PlIplom, which secured two first places. Drain was the third fastest, with one first place. The notable difference between PlIplom and Drain is that PlIplom tends to be faster than Drain in

TABLE 7

Parsing times (in seconds) on the local machine using Hadoop dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |   1/8 |   1/4 |   1/2 |   1/1 |
|----------|----------|---------|---------|---------|--------|--------|--------|-------|-------|-------|-------|
| AEL      |     0.04 |    0.02 |    0.04 |    0.07 |   0.12 |   0.2  |   0.46 |  0.79 |  1.62 |  3.24 |  6.41 |
| Brain    |     0.03 |    0.02 |    0.03 |    0.07 |   0.13 |   0.29 |   0.64 |  1.15 |  2.43 |  5.07 |  9.81 |
| Drain    |     0.01 |    0    |    0.01 |    0.01 |   0.02 |   0.03 |   0.06 |  0.12 |  0.23 |  0.47 |  0.93 |
| Iplom    |     0.01 |    0.01 |    0.01 |    0.02 |   0.04 |   0.07 |   0.13 |  0.3  |  0.66 |  1.23 |  2.29 |
| Lenma    |     0.05 |    0.04 |    0.09 |    0.2  |   0.41 |   0.86 |   1.49 |  3.41 |  7.16 | 15.73 | 29.54 |
| Pliplom  |     0.16 |    0.23 |    0.3  |    0.38 |   0.45 |   0.52 |   0.63 |  0.73 |  0.89 |  1.1  |  1.66 |
| Spell    |     0.02 |    0.02 |    0.06 |    0.14 |   0.33 |   0.82 |   1.84 |  4.5  | 10.73 | 23.04 | 47.52 |
| Tipping  |     0    |    0    |    0    |    0    |   0.01 |   0.01 |   0.02 |  0.04 |  0.07 |  0.11 |  0.22 |

TABLE 8 virtual machine using Hadoop dataset for various parsers across different dataset sizes, ranging from 1/1024

Parsing times (in seconds) on the to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |   1/8 |   1/4 |   1/2 |    1/1 |
|----------|----------|---------|---------|---------|--------|--------|--------|-------|-------|-------|--------|
| AEL      |     0.07 |    0.03 |    0.06 |    0.1  |   0.22 |   0.43 |   0.8  |  1.53 |  3.29 |  6.77 |  13.29 |
| Brain    |     0.04 |    0.04 |    0.07 |    0.15 |   0.41 |   0.51 |   1.21 |  2.17 |  4.96 | 11.15 |  21.18 |
| Drain    |     0.02 |    0.01 |    0.01 |    0.02 |   0.04 |   0.08 |   0.15 |  0.31 |  0.56 |  1.18 |   2.22 |
| Iplom    |     0.04 |    0.02 |    0.03 |    0.06 |   0.08 |   0.15 |   0.27 |  0.56 |  1.01 |  2.17 |   5.01 |
| Lenma    |     0.06 |    0.1  |    0.21 |    0.45 |   0.94 |   1.99 |   4.35 |  9.27 | 19.05 | 40.97 |  86.78 |
| PlIplom  |     0.2  |    0.3  |    0.29 |    0.45 |   0.53 |   0.66 |   0.82 |  1.02 |  1.38 |  1.86 |   2.8  |
| Spell    |     0.04 |    0.05 |    0.14 |    0.34 |   0.91 |   2.07 |   4.54 | 10.08 | 21.68 | 49.11 | 105.34 |
| Tipping  |     0.03 |    0.01 |    0.01 |    0.01 |   0.01 |   0.02 |   0.03 |  0.05 |  0.11 |  0.2  |   0.36 |

TABLE 9

Parsing times (in seconds) on the local machine using Nezha-TrainTicket dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

TABLE 10 Parsing times (in seconds) on the virtual machine using Nezha-TrainTicket dataset for various parsers across different dataset sizes, ranging

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |   1/8 |   1/4 |   1/2 |    1/1 |
|----------|----------|---------|---------|---------|--------|--------|--------|-------|-------|-------|--------|
| AEL      |     0.35 |    0.25 |    0.43 |    0.39 |   0.39 |   0.89 |   1.5  |  2.83 |  5.69 |  9.9  |  16.86 |
| Brain    |     0.04 |    0.07 |    0.1  |    0.19 |   0.4  |   0.83 |   1.62 |  3.32 |  6.19 | 22.62 |  32.58 |
| Drain    |     0    |    0.01 |    0.01 |    0.01 |   0.03 |   0.05 |   0.09 |  0.17 |  0.33 |  0.89 |   1.74 |
| Iplom    |     0.01 |    0.02 |    0.03 |    0.05 |   0.09 |   0.19 |   0.38 |  0.72 |  1.42 |  3.36 |   6.28 |
| Lenma    |     0.03 |    0.07 |    0.15 |    0.35 |   0.72 |   1.59 |   3.38 |  7.83 | 18.46 | 48.39 | 122.89 |
| Pliplom  |     0.72 |    0.76 |    0.92 |    1.09 |   1.33 |   1.76 |   2.48 |  4.4  |  5.13 |  7.88 |   9.69 |
| Spell    |     0.05 |    0.13 |    0.2  |    0.49 |   1.09 |   2    |   4.15 |  8.42 | 18.18 | 40.46 |  78.96 |
| Tipping  |     0.01 |    0.01 |    0.01 |    0.02 |   0.03 |   0.03 |   0.06 |  0.12 |  0.43 |  1.05 |   2.01 |

from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |   1/8 |   1/4 |    1/2 |    1/1 |
|----------|----------|---------|---------|---------|--------|--------|--------|-------|-------|--------|--------|
| AEL      |     0.39 |    0.44 |    0.68 |    0.72 |   0.95 |   1.8  |   2.77 |  5.63 | 11.02 |  21.25 |  37.6  |
| Brain    |     0.11 |    0.12 |    0.16 |    0.33 |   1.3  |   2.06 |   3.12 |  7.49 | 15.18 |  30.17 |  63.2  |
| Drain    |     0.05 |    0.01 |    0.02 |    0.04 |   0.07 |   0.12 |   0.24 |  0.46 |  0.89 |   1.83 |   3.7  |
| Iplom    |     0.07 |    0.05 |    0.06 |    0.11 |   0.22 |   0.42 |   0.82 |  1.69 |  2.98 |   6.43 |  13.55 |
| Lenma    |     0.17 |    0.23 |    0.46 |    1.13 |   1.98 |   4.29 |   9.25 | 21.2  | 51.9  | 135.89 | 349.02 |
| Pliplom  |     0.89 |    1.22 |    1.3  |    1.98 |   2.15 |   2.91 |   4.17 |  5.83 |  8.42 |  10.35 |  14.72 |
| Spell    |     0.22 |    0.31 |    0.45 |    1.05 |   2.13 |   4.03 |   8.38 | 17.54 | 40.17 |  83.17 | 164.07 |
| Tipping  |     0.02 |    0.02 |    0.01 |    0.02 |   0.02 |   0.04 |   0.09 |  0.18 |  0.34 |   0.62 |   1.11 |

TABLE 11 Parsing times (in seconds) on the local machine using Nezha-Shop dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |    1/8 |    1/4 |     1/2 | 1/1     |
|----------|----------|---------|---------|---------|--------|--------|--------|--------|--------|---------|---------|
| AEL      |     0.17 |    0.27 |    0.48 |    0.92 |   2.1  |   3.6  |   7.06 |  13.38 |  28.69 |   58.08 | 119.08  |
| Brain    |     0.16 |    0.33 |    0.71 |    1.35 |   2.7  |   6.03 |  12.12 |  23.45 |  46.38 |   95.54 | 191.24  |
| Drain    |     0.02 |    0.03 |    0.06 |    0.14 |   0.31 |   0.57 |   1.38 |   2.79 |   5.71 |    9.9  | 25.61   |
| Iplom    |     0.04 |    0.07 |    0.14 |    0.28 |   0.57 |   1.4  |   3.07 |   6.06 |  10.61 |   19.46 | 39.52   |
| Lenma    |     1.34 |    3.06 |    6.7  |   12.6  |  26.86 |  57.47 | 126.78 | 244.08 | 470.02 | 1660.49 | OTL     |
| Pliplom  |     0.76 |    1.33 |    1.06 |    1.2  |   1.44 |   1.47 |   1.59 |   1.89 |   2.39 |    3.07 | 4.73    |
| Spell    |     0.27 |    0.61 |    1.26 |    3.25 |   6.26 |  14.33 |  32.68 |  75.75 | 164.32 |  364.17 | 1556.72 |
| Tipping  |     0.01 |    0.02 |    0.03 |    0.07 |   0.09 |   0.24 |   0.52 |   0.99 |   2.07 |    3.62 | 7.69    |

dataset for various parsers across different dataset sizes, ranging from

TABLE 12 Parsing times (in seconds) on the virtual machine using Nezha-Shop 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |    1/8 |     1/4 | 1/2    | 1/1    |
|----------|----------|---------|---------|---------|--------|--------|--------|--------|---------|--------|--------|
| AEL      |     0.32 |    0.47 |    1.01 |    1.8  |   3.73 |   7.83 |  16.45 |  31.66 |   68.67 | 139.55 | 291.96 |
| Brain    |     0.37 |    0.57 |    1.01 |    2.08 |   4.18 |   9.65 |  20.32 |  40.82 |   91.96 | 181.47 | 357.70 |
| Drain    |     0.15 |    0.09 |    0.17 |    0.33 |   0.71 |   1.35 |   2.68 |   5.67 |   10.62 | 20.02  | 40.52  |
| Iplom    |     0.25 |    0.13 |    0.25 |    0.51 |   1.03 |   2.44 |   4.47 |  11.43 |   28.73 | 54.01  | 106.90 |
| Lenma    |     2.63 |    5.58 |   11.13 |   23.7  |  50.15 | 103.71 | 225.87 | 511.49 | 1183.57 | OTL    | OTL    |
| Pliplom  |     0.61 |    0.53 |    0.56 |    0.99 |   1.24 |   1.11 |   1.58 |   2.09 |    3.84 | 6.53   | 11.52  |
| Spell    |     0.57 |    1.16 |    2.76 |    6.86 |  15.81 |  34.6  |  79.54 | 184.76 |  417.56 | 886.77 | OTL    |
| Tipping  |     0.08 |    0.02 |    0.04 |    0.08 |   0.14 |   0.25 |   0.43 |   0.92 |    1.72 | 3.12   | 5.89   |

TABLE 13 Parsing times (in seconds) on the local machine using BGL dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

TABLE 14

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 | 1/16   | 1/8    | 1/4    | 1/2    | 1/1    |
|----------|----------|---------|---------|---------|--------|--------|--------|--------|--------|--------|--------|
| AEL      |     0.2  |    0.37 |    0.66 |    1.44 |   3.38 |   5.72 | 11.78  | 23.58  | 48.73  | 96.24  | 348.72 |
| Brain    |     0.16 |    0.31 |    0.66 |    1.25 |   3.16 |   5.38 | 11.75  | 22.68  | 44.08  | 90.17  | 169.00 |
| Drain    |     0.03 |    0.05 |    0.09 |    0.17 |   0.33 |   0.75 | 1.36   | 2.71   | 5.51   | 12.68  | 21.14  |
| Iplom    |     0.06 |    0.09 |    0.19 |    0.37 |   0.76 |   1.39 | 3.46   | 7.99   | 14.61  | 30.28  | 43.58  |
| Lenma    |     6.8  |   15.99 |   39.01 |   96.63 | 223.21 | 611.27 | OTL    | OTL    | OTL    | OTL    | OTL    |
| Pliplom  |     1.05 |    1.32 |    1.58 |    1.89 |   2.33 |   2.84 | 3.14   | 4.11   | 5.12   | 7.69   | 12.18  |
| Spell    |     0.45 |    1.19 |    2.96 |    5.81 |  13.39 |  29.93 | 68.99  | 168.33 | 424.29 | 975.84 | OTL    |
| Tipping  |     0.03 |    0.02 |    0.04 |    0.07 |   0.15 |   0.42 | 0.69   | 1.44   | 2.78   | 5.87   | 8.86   |

Parsing times (in seconds) on the virtual machine using BGL dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |    1/32 | 1/16   | 1/8    | 1/4    | 1/2    | 1/1    |
|----------|----------|---------|---------|---------|--------|---------|--------|--------|--------|--------|--------|
| AEL      |     0.4  |    0.61 |    1.23 |    2.4  |   4.5  |   10.01 | 20.36  | 42.46  | 87.72  | 177.31 | 639.22 |
| Brain    |     0.36 |    0.64 |    1.33 |    2.36 |   4.36 |   11.97 | 26.45  | 53.28  | 101.79 | 186.76 | 354.50 |
| Drain    |     0.06 |    0.08 |    0.16 |    0.34 |   0.67 |    1.35 | 2.87   | 5.61   | 11.81  | 24.08  | 44.20  |
| Iplom    |     0.18 |    0.16 |    0.32 |    0.62 |   1.11 |    2.12 | 6.19   | 12.56  | 28.00  | 57.47  | 100.47 |
| Lenma    |    12.42 |   36.92 |   98.55 |  226.25 | 522.69 | 1091.13 | OTL    | OTL    | OTL    | OTL    | OTL    |
| Pliplom  |     0.55 |    0.55 |    0.75 |    0.86 |   1.22 |    1.63 | 2.22   | 2.99   | 4.63   | 8.78   | 15.03  |
| Spell    |     0.75 |    1.59 |    3.84 |    8.56 |  20.99 |   48.27 | 110.90 | 254.86 | 608.05 | OTL    | OTL    |
| Tipping  |     0.06 |    0.03 |    0.05 |    0.08 |   0.15 |    0.27 | 0.49   | 1.00   | 1.95   | 3.90   | 7.35   |

TABLE 15

Parsing times (in seconds) on the local machine using Android dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |    1/16 | 1/8    | 1/4    | 1/2    | 1/1     |
|----------|----------|---------|---------|---------|--------|--------|---------|--------|--------|--------|---------|
| AEL      |     0.19 |    0.35 |    0.74 |    1.37 |   2.71 |   6.63 |   13.53 | 40.17  | 142.04 | 521.12 | 1657.02 |
| Brain    |     0.19 |    0.35 |    0.79 |    1.45 |   3.21 |   7.36 |   14.81 | 27.96  | 53.83  | 109.59 | 210.54  |
| Drain    |     0.04 |    0.05 |    0.09 |    0.18 |   0.51 |   0.95 |    2.02 | 4.65   | 7.96   | 15.79  | 30.56   |
| Iplom    |     0.06 |    0.09 |    0.19 |    0.41 |   0.8  |   1.75 |    4.5  | 8.35   | 16.41  | 31.51  | 57.86   |
| Lenma    |     2.12 |    5.84 |   14.26 |   31.2  |  73.15 | 174.46 |  386.23 | 803.56 | OTL    | OTL    | OTL     |
| Pliplom  |     1.88 |    3.12 |    4.2  |    5.93 |   7.78 |  10.62 |   12.56 | 16.02  | 20.70  | 26.13  | 33.04   |
| Spell    |     2.66 |    6.18 |   23.36 |   56.15 | 150.84 | 416.88 | 1096.78 | OTL    | OTL    | OTL    | OTL     |
| Tipping  |     0.04 |    0.02 |    0.05 |    0.09 |   0.16 |   0.32 |    0.65 | 1.37   | 2.73   | 5.19   | 9.44    |

TABLE 16 Parsing times (in seconds) on the virtual machine using Android dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

TABLE 17 Parsing times (in seconds) on the local machine using HDFS dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |    1/16 | 1/8   | 1/4    | 1/2     | 1/1    |
|----------|----------|---------|---------|---------|--------|--------|---------|-------|--------|---------|--------|
| AEL      |     0.58 |    0.66 |    1.81 |    3.2  |   6.28 |  12.39 |   29.96 | 79.40 | 230.08 | 1268.74 | OTL    |
| Brain    |     0.48 |    0.7  |    1.39 |    2.72 |   6.87 |  13.14 |   29.47 | 61.62 | 118.08 | 243.72  | 468.88 |
| Drain    |     0.12 |    0.12 |    0.24 |    0.43 |   0.84 |   1.74 |    4.08 | 7.24  | 14.90  | 31.40   | 60.84  |
| Fiplom   |    14.92 |   21.72 |   32.2  |   43.48 |  55.78 |  65.95 |   77.89 | 98.67 | 118.54 | 145.41  | 170.02 |
| Iplom    |     0.18 |    0.17 |    0.44 |    0.85 |   1.56 |   3.49 |    7.75 | 15.55 | 32.01  | 62.45   | 119.40 |
| Lenma    |     3.67 |    8.15 |   19.64 |   47.01 | 112.64 | 264.75 |  606.25 | OTL   | OTL    | OTL     | OTL    |
| Pliplom  |     1.64 |    2.65 |    3.51 |    4.88 |   6.15 |   7.76 |   10.46 | 13.12 | 18.61  | 24.98   | 37.50  |
| Spell    |     3.35 |   10.2  |   27.17 |   72.79 | 201.45 | 514.25 | 1313.77 | OTL   | OTL    | OTL     | OTL    |
| Tipping  |     0.15 |    0.04 |    0.06 |    0.12 |   0.19 |   0.34 |    0.57 | 1.16  | 2.15   | 4.49    | 8.82   |

TABLE 18

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |    1/8 |    1/4 | 1/2    | 1/1    |
|----------|----------|---------|---------|---------|--------|--------|--------|--------|--------|--------|--------|
| AEL      |     0.43 |    0.65 |    1.43 |    3.43 |   7.13 |  10.94 |  20.66 |  40.94 |  80.88 | 165.51 | 338.08 |
| Brain    |     0.76 |    1.13 |    1.89 |    4.64 |   6.63 |  19.56 |  34.24 |  63.73 | 128.22 | 255.77 | DNF    |
| Drain    |     0.08 |    0.22 |    0.21 |    0.42 |   0.85 |   1.8  |   4.21 |   9.11 |  14.81 | 29.80  | 76.20  |
| Iplom    |     0.23 |    0.29 |    0.87 |    1.39 |   3.11 |   4.98 |  10.23 |  19.07 |  32.74 | 64.23  | 119.42 |
| Lenma    |     3.24 |    6.56 |   11.94 |   19.69 |  37.69 |  75.23 | 151.22 | 301.56 | 612.81 | OTL    | OTL    |
| Pliplom  |     0.13 |    0.1  |    0.15 |    0.19 |   0.29 |   0.5  |   0.87 |   1.6  |   3.04 | 5.05   | 8.89   |
| Spell    |     0.5  |    1.01 |    2.08 |    4.54 |   9.39 |  19.95 |  43.19 |  93.35 | 204.9  | 410.00 | 821.56 |
| Tipping  |     0.05 |    0.05 |    0.06 |    0.09 |   0.17 |   0.39 |   0.95 |   1.98 |   3.99 | 9.38   | 16.44  |

Parsing times (in seconds) on the virtual machine

using

HDFS

dataset for various parsers across different dataset sizes, ranging from 1/1024 to the full dataset.

| Method   |   1/1024 |   1/512 |   1/256 |   1/128 |   1/64 |   1/32 |   1/16 |    1/8 | 1/4    | 1/2    | 1/1     |
|----------|----------|---------|---------|---------|--------|--------|--------|--------|--------|--------|---------|
| AEL      |     1.05 |    1.55 |    3.05 |    5.93 |  11.08 |  22.5  |  45.16 |  90.46 | 182.24 | 358.16 | 727.55  |
| Brain    |     0.99 |    1.51 |    3.32 |    6.08 |  12.31 |  26.61 |  62.69 | 122.65 | 246.63 | 478.24 | 1016.13 |
| Drain    |     0.17 |    0.27 |    0.5  |    1.01 |   1.65 |   3.39 |   6.47 |  13.33 | 28.02  | 58.52  | 128.90  |
| Iplom    |     0.48 |    0.52 |    0.97 |    1.74 |   3.43 |   7.17 |  14.89 |  37.84 | 70.83  | 142.33 | 256.24  |
| Lenma    |     5.1  |    9.83 |   20.42 |   39.47 |  82.55 | 168.56 | 335.38 | 687.35 | OTL    | OTL    | OTL     |
| Pliplom  |     0.26 |    0.19 |    0.26 |    0.43 |   0.58 |   1.07 |   1.95 |   4.3  | 8.62   | 16.33  | 31.21   |
| Spell    |     0.94 |    1.47 |    3.17 |    6.67 |  15.57 |  32.42 |  69.31 | 144.91 | 328.61 | 697.07 | OTL     |
| Tipping  |     0.22 |    0.05 |    0.1  |    0.19 |   0.34 |   0.64 |   1.06 |   2.26 | 4.71   | 8.84   | 16.60   |

TABLE 19

Parsing times (in seconds) on the virtual machine using Thunderbird dataset for various parsers across different dataset sizes, ranging from 1/1024 to 1/2 of the dataset.

| Method   |   1/1024 |   1/512 | 1/256   | 1/128   | 1/64   | 1/32   | 1/16    | 1/8    | 1/4    | 1/2    |
|----------|----------|---------|---------|---------|--------|--------|---------|--------|--------|--------|
| AEL      |    13.49 |   25.74 | 56.54   | 113.87  | 226.90 | 491.41 | 1013.88 | OTL    | OTL    | OTL    |
| Brain    |    17.31 |   37.63 | 88.18   | 179.87  | 350.80 | 640.34 | OTL     | OTL    | OTL    | OTL    |
| Drain    |     2.48 |    5.18 | 11.45   | 24.03   | 45.45  | 100.54 | 195.61  | 386.25 | 800.11 | OTL    |
| Iplom    |     4.87 |    7.63 | 16.31   | 48.33   | 99.79  | 200.76 | 396.21  | 963.39 | OTL    | OTL    |
| Lenma    |   287.93 |  638.99 | OTL     | OTL     | OTL    | OTL    | OTL     | OTL    | OTL    | OTL    |
| Pliplom  |     3.73 |    5.8  | 7.33    | 12.44   | 20.81  | 34.72  | 74.82   | 142.60 | 322.63 | 720.07 |
| Spell    |   446.08 | 1028.88 | OTL     | OTL     | OTL    | OTL    | OTL     | OTL    | OTL    | OTL    |
| Tipping  |     0.97 |    0.91 | 1.62    | 3.44    | 6.46   | 12.64  | 26.88   | 83.50  | 174.77 | 292.95 |

TABLE 20

Parsing times (in seconds) on the virtual machine using Spirit dataset for various parsers across different dataset sizes, ranging from 1/1024 to 1/4 of the dataset.

| Method   |   1/1024 |   1/512 | 1/256   | 1/128   | 1/64   | 1/32    | 1/16   | 1/8     | 1/4    |
|----------|----------|---------|---------|---------|--------|---------|--------|---------|--------|
| AEL      |    18.14 |   35.69 | 72.63   | 146.75  | 326.59 | 671.77  | OTL    | OTL     | OTL    |
| Brain    |    27.04 |   56.41 | 126.17  | 253.24  | 556.99 | 1119.26 | OTL    | OTL     | OTL    |
| Drain    |     3.37 |    7.32 | 13.93   | 27.02   | 61.53  | 122.03  | 235.50 | 411.58  | DNF    |
| Fiplom   |    69.21 |   93.98 | 149.81  | 265.75  | 484.10 | 869.52  | OTL    | OTL     | OTL    |
| Iplom    |     5.79 |   10.68 | 31.64   | 65.18   | 138.55 | 291.27  | 514.12 | 1100.10 | OTL    |
| Lenma    |   207.35 |  497.37 | 1487.78 | OTL     | OTL    | OTL     | OTL    | OTL     | OTL    |
| Pliplom  |     3.45 |    5.57 | 11.14   | 19.48   | 38.72  | 72.27   | 170.26 | 378.54  | 984.92 |
| Spell    |   417.94 | 1006.59 | OTL     | OTL     | OTL    | OTL     | OTL    | OTL     | OTL    |
| Tipping  |     1.38 |    1.11 | 2.09    | 3.79    | 7.56   | 16.85   | 31.08  | 104.33  | 119.07 |

Parsing times (in seconds) on the virtual machine using Liberty dataset for various parsers across different dataset sizes, ranging from 1/1024 to

TABLE 21 1/2 of the dataset.

| Parser   |   1/1024 |   1/512 |   1/256 | 1/128   | 1/64   | 1/32   | 1/16    | 1/8    | 1/4    | 1/2    |
|----------|----------|---------|---------|---------|--------|--------|---------|--------|--------|--------|
| AEL      |    18.11 |   32.47 |   66.77 | 130.24  | 265.87 | 553.11 | 1247.87 | OTL    | OTL    | OTL    |
| Brain    |    17.67 |   35.54 |   77.04 | 168.11  | 350.12 | 675.11 | OTL     | OTL    | OTL    | OTL    |
| Drain    |     2.37 |    4.99 |   10.1  | 21.48   | 46.38  | 98.69  | 182.59  | 347.35 | 707.99 | OTL    |
| Fiplom   |    38.95 |   59.5  |   84.1  | 138.30  | 213.22 | 369.22 | 658.94  | OTL    | OTL    | OTL    |
| Iplom    |     4.27 |    7.77 |   15.96 | 47.13   | 90.16  | 185.26 | 378.99  | 824.79 | OTL    | OTL    |
| Lenma    |   149.24 |  385.89 |  999.13 | OTL     | OTL    | OTL    | OTL     | OTL    | OTL    | OTL    |
| Pliplom  |     2.2  |    3.59 |    5.9  | 11.96   | 23.17  | 39.74  | 80.59   | 171.82 | 230.52 | 478.18 |
| Spell    |   218.75 |  563.7  | 1503.56 | OTL     | OTL    | OTL    | OTL     | OTL    | OTL    | OTL    |
| Tipping  |     0.8  |    0.81 |    1.85 | 3.55    | 6.20   | 12.57  | 24.62   | 51.10  | 116.22 | DNF    |

medium and large datasets, while Drain performs better in small datasets.

Investigating Tipping's parallelization capability further, we compared how Tipping performs with itself when run on the same dataset in LM and VM. Remember that VM has multiple, slower processors, while LM has fewer but faster processors. We found that Tipping is faster in the VM in four of the six datasets. In the HDFS dataset, Tipping is faster in the LM by a small margin of 1%. Furthermore, Tipping performs better in the LM on the Hadoop dataset, which is considered a small dataset with parsing times less than 0.5 seconds, indicating that parallelization does not pay off in such small datasets.

The fact that Tipping is faster in the VM distinguishes it from all other parsers, as all other parsers consistently show slower performance in the VM compared to the LM. For example, Drain, which has been the benchmark parser for years and was the overall third fastest parser in our tests, shows significantly slower performance in the VM across various datasets. Specifically, Drain is 139%, 126%, 58%, 110%, 99%, and 69% slower in the VM than in the LM

for the Hadoop, TrainTicket, Web-shop, BGL, Android, and HDFS datasets, respectively. This contrast underscores Tipping's superior parallelization capabilities when deployed in environments with more CPU cores.

## 4.4 Anomaly Detection benchmark with LogLead

## 4.4.1 Experiment depiction

In our final experiment, we examine how the parsing output of different parsers affects the downstream task of anomaly detection in software logs. Past work provides somewhat mixed results on whether the accuracy of log parsing impacts the performance of anomaly detection [42]. Nevertheless, in a real-world scenario, the accuracy of the end goals is an important property of analysis pipelines, as ideally, one chooses components based on parsing performance and the end goal's accuracy.

We used 3 datasets that we were able to fully parse: Hadoop, HDFS, and BGL. The remaining supercomputer datasets, Thunderbird, Liberty, and Spirit, were excluded due to their size. Trainticket and Web-shop are multimodal

datasets consisting of not only logs but also traces and metrics, which must be analyzed to achieve decent performance on those datasets. Finally, our Android2 dataset contained only 2 anomaly instances, making it impossible to use for anomaly detection investigations.

We tested the top three fastest parsers from the previous experiment: Tipping, PlIpLom, and Drain alongside Brain, as it had shown good accuracy in our first and second experiments.

We tested a combination of supervised and unsupervised algorithms to determine if parsing results impact anomaly detection accuracy. We used the Decision Tree (su-DT), Linear Support-Vector Machine (su-SVM), Logistic Regression (su-LR), Random Forest (su-RF), and Extreme Gradient Boosting Tree (su-XGB) for supervised methods, and K-means clustering (us-KM) and Isolation Forest (usIF) for unsupervised methods. Additionally, we tested a custom-made unsupervised model called RarityModel (usRM) [43] that utilizes the negative logarithm to measure the rarity of individual log events. This model proposes that the rarer a log event, the higher the likelihood of an anomaly.

We used two accuracy measures: F 1 score and AUCROC. The F 1 score indicates how well the model detects anomalies with a fixed (optimal) threshold. The AUC-ROC score shows how well the model detects anomalies if one considers all possible thresholds. The AUC-ROC score is important for practical usage, as higher precision or recall may be required in certain use cases.

Our training and testing procedure varied between datasets as they are very different in nature. HDFS and Hadoop are experimental datasets, meaning that researchers had run the systems in a lab while injecting anomalies in an arbitrary order. Thus, there is no real-world chronological order, and sequences from those datasets can be placed in random order when testing anomaly detection performance. However, random order cannot be used for data originating from real-world usage, as one cannot use future data to predict past performance. This is the case for BGL, which comes from production use over 200 days.

For Hadoop, we randomized data, split the data to 50-50 for training and testing, repeated the procedure 100 times, and computed the average of F 1 score and AUC-ROC for each log parser and machine learning algorithm. Hadoop is a fairly small dataset, so a 50-50 split was needed to get decent performance. HDFS, however, suffers from the opposite problem, i.e., results tend to be too good if the entire dataset is used. Therefore, we used only random 1% of data and repeated this procedure 100 times. For each 1%, we did ten random splits while using 10% data for training and 90% for testing. This resulted in a far lower F 1 score than prior works but allowed us to better investigate parser differences. For BGL, we maintained chronological order and tested 10 different train-test splits (5-95, 15-85, 25-75, 35-65, 45-55, 55-45, 65-35, 75-25, 85-15, 95-5) and computed the average.

## 4.4.2 Experiment results

Hadoop results are presented in Tables 22 and 23. PlIpLom achieves the best results in terms of both F 1 score and AUC-ROC, followed by Tipping, while the last place alternates between Drain and Brain. The difference between

TABLE 22 AUC-ROC of anomaly detection on Hadoop

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.692 |   0.703 |     0.697 |     0.716 |  0.702 |
| su-SVM  |   0.74  |   0.74  |     0.825 |     0.768 |  0.768 |
| su-LR   |   0.735 |   0.733 |     0.745 |     0.748 |  0.74  |
| su-RF   |   0.836 |   0.835 |     0.842 |     0.839 |  0.838 |
| su-XGB  |   0.808 |   0.807 |     0.829 |     0.825 |  0.817 |
| us-IF   |   0.283 |   0.276 |     0.267 |     0.287 |  0.278 |
| us-KM   |   0.442 |   0.432 |     0.442 |     0.443 |  0.44  |
| us-RM   |   0.242 |   0.243 |     0.244 |     0.25  |  0.245 |
| Avg.    |   0.597 |   0.596 |     0.612 |     0.61  |  0.604 |

TABLE 23 F 1 score of anomaly detection on Hadoop

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.464 |   0.489 |     0.47  |     0.503 |  0.481 |
| su-SVM  |   0.383 |   0.377 |     0.574 |     0.431 |  0.441 |
| su-LR   |   0.396 |   0.398 |     0.409 |     0.401 |  0.401 |
| su-RF   |   0.414 |   0.44  |     0.432 |     0.445 |  0.433 |
| su-XGB  |   0.458 |   0.461 |     0.459 |     0.477 |  0.464 |
| us-IF   |   0.056 |   0.08  |     0.072 |     0.106 |  0.078 |
| us-KM   |   0     |   0     |     0     |     0     |  0     |
| us-RM   |   0.335 |   0.335 |     0.335 |     0.335 |  0.335 |
| Avg.    |   0.313 |   0.322 |     0.344 |     0.337 |  0.329 |

the winner, PlIpLom, and the last parser in the AUC-ROC average is 0.016 (0.612 - 0.596) and in F 1 score 0.024 (0.337 -0.313). PlIpLom's top performance stems from its results with the SVM model, where it significantly outperforms the last parser with an AUC-ROC of 0.825 compared to 0.750 and an F 1 score of 0.574 compared to 0.377. The differences between PlIpLom and other parsers are negligible with all other machine learning models.

Tables 24 and 25 demonstrate the HDFS results, showing no meaningful differences between parsers' average F 1 scores or AUC-ROCs. However, when examining individual machine learning models and parsers pairs, differences in terms of F 1 score are observed (see Table 25). Brain shows slightly weaker performance with LR and RF compared to the rest but compensates with better performance using the IF model. The notable differences between machine learning algorithms suggest that the setup should allow for detecting differences between parsers if there are any.

BGL results are in Tables 26 and 27. Tipping is the winner while the margin to the last parser in AUC-ROC is 0.068

TABLE 24 AUC-ROC of anomaly detection on HDFS

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.95  |   0.951 |     0.951 |     0.951 |  0.951 |
| su-SVM  |   0.944 |   0.947 |     0.946 |     0.947 |  0.946 |
| su-LR   |   0.947 |   0.946 |     0.946 |     0.947 |  0.947 |
| su-RF   |   0.975 |   0.975 |     0.975 |     0.975 |  0.975 |
| su-XGB  |   0.915 |   0.915 |     0.915 |     0.915 |  0.915 |
| us-IF   |   0.951 |   0.951 |     0.951 |     0.951 |  0.951 |
| us-KM   |   0.842 |   0.843 |     0.843 |     0.844 |  0.843 |
| us-RM   |   0.608 |   0.608 |     0.608 |     0.608 |  0.608 |
| Avg.    |   0.891 |   0.892 |     0.892 |     0.892 |  0.892 |

TABLE 25 F 1 score of anomaly detection on HDFS

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.928 |   0.929 |     0.929 |     0.929 |  0.929 |
| su-SVM  |   0.9   |   0.902 |     0.902 |     0.902 |  0.901 |
| su-LR   |   0.644 |   0.647 |     0.647 |     0.648 |  0.646 |
| su-RF   |   0.897 |   0.899 |     0.9   |     0.9   |  0.899 |
| su-XGB  |   0.836 |   0.834 |     0.836 |     0.836 |  0.836 |
| us-IF   |   0.438 |   0.434 |     0.429 |     0.429 |  0.433 |
| us-KM   |   0     |   0     |     0     |     0     |  0     |
| us-RM   |   0.122 |   0.125 |     0.125 |     0.125 |  0.124 |
| Avg.    |   0.596 |   0.596 |     0.596 |     0.596 |  0.596 |

TABLE 26 AUC-ROC of anomaly detection on BGL

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.638 |   0.639 |     0.639 |     0.749 |  0.667 |
| su-SVM  |   0.842 |   0.837 |     0.844 |     0.907 |  0.858 |
| su-LR   |   0.842 |   0.841 |     0.845 |     0.908 |  0.859 |
| su-RF   |   0.638 |   0.639 |     0.639 |     0.782 |  0.675 |
| su-XGB  |   0.793 |   0.789 |     0.797 |     0.854 |  0.808 |
| us-IF   |   0.334 |   0.396 |     0.364 |     0.4   |  0.373 |
| us-KM   |   0.541 |   0.457 |     0.491 |     0.541 |  0.507 |
| us-RM   |   0.451 |   0.425 |     0.425 |     0.426 |  0.432 |
| Avg.    |   0.635 |   0.628 |     0.631 |     0.696 |  0.647 |

(0.696 - 0.628) and in F 1 score 0.104 (0.329 - 0.225). Tipping is superior with all supervised algorithms and IF from the unsupervised algorithms. With unsupervised algorithms, Brain is better than Tipping with RM and is on par with KM.

## 4.4.3 Summary

We received mixed results when testing four parsers in three datasets with 8 different algorithms. For the HDFS dataset, there was no difference between parsers, resulting in an astonishingly equal four-way tie. In Hadoop, PlIplon was the best, but its victory mainly stemmed from being widely superior with one algorithm (SVM), while with other machine learning models, the differences were negligible. In BGL, Tipping was superior with all supervised models and one unsupervised model, but with two unsupervised models, Tipping was no better than the others.

Past work [42] has proposed that the number of templates a parser generates is negatively correlated with the anomaly detection capability of the parser. We checked the

TABLE 27 F 1 score of anomaly detection on BGL

| Model   |   Brain |   Drain |   PlIpLom |   Tipping |   Avg. |
|---------|---------|---------|-----------|-----------|--------|
| su-DT   |   0.373 |   0.377 |     0.375 |     0.559 |  0.421 |
| su-SVM  |   0.336 |   0.377 |     0.375 |     0.559 |  0.412 |
| su-LR   |   0.339 |   0.343 |     0.339 |     0.483 |  0.376 |
| su-RF   |   0.373 |   0.377 |     0.375 |     0.559 |  0.421 |
| su-XGB  |   0.332 |   0.337 |     0.333 |     0.463 |  0.366 |
| us-IF   |   0.002 |   0.002 |     0     |     0     |  0.001 |
| us-KM   |   0     |   0     |     0     |     0     |  0     |
| us-RM   |   0.047 |   0.01  |     0.011 |     0.012 |  0.02  |
| Avg.    |   0.225 |   0.228 |     0.226 |     0.329 |  0.252 |

TABLE 28 Number of events produced

| Dataset   |   Brain |   Drain |   PlIpLom |   Tip | [44]   | [1]   |
|-----------|---------|---------|-----------|-------|--------|-------|
| Hadoop    |     306 |     292 |       267 |   233 | 312*   | 253   |
| HDFS      |      44 |      53 |        46 |    37 | 33     | 53    |
| BGL       |     643 |     850 |      1018 |   302 | 394    | NA    |

* We subtracted event #37 that described stack traces

number of events the four parsers produced, as shown in Table 28, and found mixed results. In HDFS, Drain produces 43% more events than Tipping, but there is no difference in anomaly detection capability in that dataset. In Hadoop, PlIplom was the best parser, but it did not produce the smallest amount of events. This theory only holds in the BGL dataset, where Tipping produced the smallest number of events and was the best parser.

Table 28 also compares the 'correct' number of templates from two sources produced by humans on the full dataset [1], [44]. Event numbers from the sources do not match each other, indicating that it is hard to arrive at the same result, even when we agree on the concept of templates and parameters on a higher level. Using those numbers suggests that all parsers are in the right ballpark regarding HDFS and Hadoop. However, regarding BGL, many parsers seem to produce too many templates. Thus, this would bring forward a new hypothesis that the number of templates decreases the anomaly detection accuracy only if the number deviates too much from the 'correct' number of templates.

## 5 THREATS TO VALIDITY

## 5.1 Internal Validity

Our experiments were conducted on specific hardware configurations, including machines with multiple CPU cores. The performance gap between Tipping and other parsers might not be as wide as our experiment under different hardware settings, especially when the target machine lacks in quantity of CPU cores. However, this situation seems not to be prevalent in today's age of technology.

We allocated no more than five minutes to parameter tuning per dataset in experiments in Sections 4.1 and 4.2. However, it might not reflect a realistic timeframe, as the authors are knowledgeable and familiar with both the datasets and the algorithm.

## 5.2 External Validity

The datasets utilized in our study were chosen to represent a broad range of real-world software systems. However, the open public datasets used are not sourced from the software's latest version and may not encompass all possibilities. Furthermore, our experiments were conducted in controlled environments, which may not perfectly replicate the complexities and variabilities of real-world log data. Therefore, while our results are indicative of Tipping's performance across diverse conditions, they may not fully generalize to all log parsing contexts.

## 6 CONCLUSION

In this paper, we introduced a novel log parsing algorithm named 'Tipping,' which utilizes a combination of rule-based tokenizers, interdependency token graphs, and strongly connected components to parse logs. Our comprehensive evaluation across various benchmarks demonstrates that Tipping consistently outperforms existing stateof-the-art parsers in both parsing accuracy and processing speed. In the LogPai benchmark, Tipping achieved high group accuracy and F 1 scores across diverse datasets, indicating its robustness and versatility. In the LogPM benchmark, Tipping exhibited superior parsing accuracy, parameter mask agreement, and impressive parsing speeds, highlighting its efficiency and precision.

Through experiments conducted using the LogLead tool, Tipping showcased its exceptional ability to handle large datasets efficiently, demonstrating robust parallelization capabilities. It emerged as the fastest parser in the majority of scenarios, particularly excelling in environments with additional CPU cores. This computational efficiency underscores Tipping's scalability, making it a highly viable solution for large-scale log parsing tasks.

Moreover, Tipping's parsing output enhances the downstream task of anomaly detection. Although the differences in anomaly detection performance among various parsers were dataset-dependent, Tipping generally performed well, especially when integrated with supervised learning models. This consistency and adaptability underline Tipping's effectiveness as a log-parsing solution. By offering improvements over existing methods and adaptability to different environments, Tipping proves to be a valuable tool for modern log analysis and anomaly detection tasks.

## 7 BENCHMARK AND TOOL AVAILABILITY

For the sake of transparency and accountability, we opensource all repositories and packages used in this paper, including Tipping Rust 14 , Tipping Python 15 , Tipping Pip 16 LogPai 17 , LogPM 18 , and LogLead 19 implementations.

## 8 ACKNOWLEDGEMENTS

The authors have been supported by the Research Council of Finland (grant IDs: 328058, 349487, 359861)

## REFERENCES

- [1] S. Hashemi, J. Nyyss¨ ol¨ a, and M. M¨ antyl¨ a, 'Logpm: Characterbased log parser,' in Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , (Rovaniemi, Finland), pp. 705-709, IEEE, 2024.
- [2] S. Hashemi and M. M¨ antyl¨ a, 'Sialog: detecting anomalies in software execution logs using the siamese network,' Automated Software Engineering , vol. 29, p. 61, Oct 2022.
- 14. https://github.com/shshemi/tipping-rs
- 15. https://github.com/shshemi/tipping
- 16. https://pypi.org/project/tipping/
- 17. https://github.com/M3SOulu/logparser
- 18. https://github.com/M3SOulu/LogPMBenchmark
- 19. https://github.com/EvoTestOps/LogLead
- [3] W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun, et al. , 'Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,' in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization , vol. 7, pp. 4739-4745, 2019.
- [4] M. Du, F. Li, G. Zheng, and V. Srikumar, 'Deeplog: Anomaly detection and diagnosis from system logs through deep learning,' in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , pp. 1285-1298, 2017.
- [5] S. Lu, X. Wei, Y. Li, and L. Wang, 'Detecting anomaly in big data system logs using convolutional neural network,' in 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech) , pp. 151-158, IEEE, 2018.
- [6] M. Steidl, M. Gattringer, M. Felderer, R. Ramler, and M. Shahriari, 'Requirements for anomaly detection techniques for microservices,' in Product-Focused Software Process Improvement (D. Taibi, M. Kuhrmann, T. Mikkonen, J. Kl¨ under, and P. Abrahamsson, eds.), (Cham), pp. 37-52, Springer International Publishing, 2022.
- [7] L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline, and G. Venolia, 'Debugging revisited: Toward understanding the debugging needs of contemporary software developers,' in 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement , pp. 383-392, 2013.
- [8] A. R. Chen, 'An empirical study on leveraging logs for debugging production failures,' in 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSECompanion) , pp. 126-128, 2019.
- [9] B. Tierney, W. Johnston, B. Crowley, G. Hoo, C. Brooks, and D. Gunter, 'The netlogger methodology for high performance distributed systems performance analysis,' in Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244) , pp. 260-267, 1998.
- [10] C. Kabir, S. Hoadley, Q. Al-Dashti, D. Kamal, A. Hasan, and D. Lin, 'Combined production logging, transient testing, and wellperformance analysis,' in SPE Annual Technical Conference and Exhibition , OnePetro, 1998.
- [11] J. Svacina, J. Raffety, C. Woodahl, B. Stone, T. Cerny, M. Bures, D. Shin, K. Frajtak, and P. Tisnovsky, 'On vulnerability and security log analysis: A systematic literature review on recent trends,' in Proceedings of the International Conference on Research in Adaptive and Convergent Systems , pp. 175-180, 2020.
- [12] F. Skopik, M. Wurzenberger, and M. Landauer, Smart Log Data Analytics: Techniques for Advanced Security Analysis . Springer, 2021.
- [13] B. Azvine, Z. Cui, and D. D. Nauck, 'Towards real-time business intelligence,' BT Technology Journal , vol. 23, no. 3, pp. 214-225, 2005.
- [14] H. Zawawy, K. Kontogiannis, and J. Mylopoulos, 'Log filtering and interpretation for root cause analysis,' in 2010 IEEE International Conference on Software Maintenance , pp. 1-5, 2010.
- [15] S. Suriadi, C. Ouyang, W. M. P. van der Aalst, and A. H. M. ter Hofstede, 'Root cause analysis with enriched process logs,' in Business Process Management Workshops (M. La Rosa and P. Soffer, eds.), (Berlin, Heidelberg), pp. 174-186, Springer Berlin Heidelberg, 2013.
- [16] D. T. Wu, N. Smart, E. L. Ciemins, H. J. Lanham, C. Lindberg, and K. Zheng, 'Using ehr audit trail logs to analyze clinical workflow: a case study from community-based ambulatory clinics,' in AMIA Annual Symposium Proceedings , vol. 2017, p. 1820, American Medical Informatics Association, 2017.
- [17] S. Yu, P. He, N. Chen, and Y. Wu, 'Brain: Log parsing with bidirectional parallel tree,' IEEE Transactions on Services Computing , vol. 16, no. 5, pp. 3224-3237, 2023.
- [18] P. He, J. Zhu, Z. Zheng, and M. R. Lyu, 'Drain: An online log parsing approach with fixed depth tree,' in 2017 IEEE International Conference on Web Services (ICWS) , pp. 33-40, 2017.
- [19] M. Du and F. Li, 'Spell: Streaming parsing of system event logs,' in 2016 IEEE 16th International Conference on Data Mining (ICDM) , pp. 859-864, 2016.
- [20] S. Messaoudi, A. Panichella, D. Bianculli, L. Briand, and R. Sasnauskas, 'A search-based approach for accurate identification of log message formats,' in Proceedings of the 26th Conference on Program Comprehension , pp. 167-177, 2018.

- [21] H. Dai, H. Li, C. S. Chen, W. Shang, and T.-H. Chen, 'Logram: Efficient log parsing using n-gram dictionaries,' IEEE Transactions on Software Engineering , 2020.
- [22] I. Sedki, A. Hamou-Lhadj, O. Ait-Mohamed, and M. A. Shehab, 'An effective approach for parsing large log files,' in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME) , pp. 1-12, 2022.
- [23] S. He, J. Zhu, P. He, and M. R. Lyu, 'Loghub: A large collection of system log datasets towards automated log analytics,' 2020.
- [24] J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, 'Tools and benchmarks for automated log parsing,' in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , pp. 121-130, IEEE, 2019.
- [25] S. Nedelkoski, J. Bogatinovski, A. Acker, J. Cardoso, and O. Kao, 'Self-supervised log parsing,' in Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track (Y. Dong, D. Mladeni´ c, and C. Saunders, eds.), (Cham), pp. 122-138, Springer International Publishing, 2021.
- [26] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, 'A fast and elitist multiobjective genetic algorithm: Nsga-ii,' IEEE transactions on evolutionary computation , vol. 6, no. 2, pp. 182-197, 2002.
- [27] K. Shima, 'Length matters: Clustering system log messages using length of words,' arXiv preprint arXiv:1611.03213 , 2016.
- [28] A. A. Makanju, A. N. Zincir-Heywood, and E. E. Milios, 'Clustering event logs using iterative partitioning,' in Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pp. 1255-1264, 2009.
- [29] A. Makanju, A. N. Zincir-Heywood, and E. E. Milios, 'A lightweight algorithm for message type extraction in system application logs,' IEEE Transactions on Knowledge and Data Engineering , vol. 24, no. 11, pp. 1921-1936, 2011.
- [30] Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora, 'An automated approach for abstracting execution logs to execution events,' Journal of Software Maintenance and Evolution: Research and Practice , vol. 20, no. 4, pp. 249-267, 2008.
- [31] J. Lu, D. Wu, M. Mao, W. Wang, and G. Zhang, 'Recommender system application developments: a survey,' Decision support systems , vol. 74, pp. 12-32, 2015.
- [32] R. Tarjan, 'Depth-first search and linear graph algorithms,' SIAM journal on computing , vol. 1, no. 2, pp. 146-160, 1972.
- [33] M. Sharir, 'A strong-connectivity algorithm and its applications in data flow analysis,' Computers &amp; Mathematics with Applications , vol. 7, no. 1, pp. 67-72, 1981.
- [34] S. Hashemi, 'LogPM Benchmark,' Jan 2023. https://github.com/M3SOulu/LogPMBenchmark.
- [35] M. M¨ antyl¨ a, Y. Wang, and J. Nyyss¨ ol¨ a, 'Loglead - fast and integrated log loader, enhancer, and anomaly detector,' in Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , (Rovaniemi, Finland), pp. 395-399, IEEE, 2024. https://github.com/EvoTestOps/LogLead.
- [36] A. Oliner and J. Stearley, 'What supercomputers say: A study of five system logs,' in 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) , pp. 575-584, 2007.
- [37] Q. Lin, H. Zhang, J. Lou, Y. Zhang, and X. Chen, 'Log clustering based problem identification for online service systems,' in 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C) , pp. 102-111, 2016.
- [38] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, 'Detecting large-scale system problems by mining console logs,' in Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles , pp. 117-132, 2009.
- [39] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, W. Li, and D. Ding, 'Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study,' IEEE Transactions on Software Engineering , vol. 47, no. 2, pp. 243-260, 2018.
- [40] G. Yu, P. Chen, Y. Li, H. Chen, X. Li, and Z. Zheng, 'Nezha: Interpretable fine-grained root causes analysis for microservices on multi-modal observability data,' in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pp. 553-565, 2023.
- [41] M. M¨ antyl¨ a, M. Varela, and S. Hashemi, 'Pinpointing anomaly events in logs from stability testing-n-grams vs. deep-learning,' in 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) , pp. 285-292, IEEE, 2022.
- [42] Y. Fu, M. Yan, Z. Xu, X. Xia, X. Zhang, and D. Yang, 'An empirical study of the impact of log parsers on the performance of log-based
- anomaly detection,' Empirical Software Engineering , vol. 28, no. 1, pp. 1-39, 2023.
- [43] J. Nyyss¨ ol¨ a and M. M¨ antyl¨, a 'Efficiency of unsupervised anomaly detection methods on software logs,' arXiv preprint arXiv:2312.01934 , 2023.
- [44] M. Landauer, F. Skopik, and M. Wurzenberger, 'A critical review of common log data sets used for evaluation of sequence-based anomaly detection techniques,' arXiv preprint arXiv:2309.02854 , 2023.
Shayan Hashemi Mika Mäntylä
Token Interdependency Parsing (Tipping) -- Fast and Accurate Log Parsing
In the last decade, an impressive increase in software adaptions has led to a
surge in log data production, making manual log analysis impractical and
establishing the necessity for automated methods. Conversely, most automated
analysis tools include a component designed to separate log templates from
their parameters, commonly referred to as a "log parser". This paper aims to
introduce a new fast and accurate log parser, named "Tipping". Tipping combines
rule-based tokenizers, interdependency token graphs, strongly connected
components, and various techniques to ensure rapid, scalable, and precise log
parsing. Furthermore, Tipping is parallelized and capable of running on
multiple processing cores with close to linear efficiency. We evaluated Tipping
against other state-of-the-art log parsers in terms of accuracy, performance,
and the downstream task of anomaly detection. Accordingly, we found that
Tipping outperformed existing methods in accuracy and performance in our
evaluations. More in-depth, Tipping can parse 11 million lines of logs in less
than 20 seconds on a laptop machine. Furthermore, we re-implemented a
parallelized version of the past IpLom algorithm to demonstrate the effect of
parallel processing, and it became the second-fastest parser. As logs keep
growing in volume and complexity, the software engineering community needs to
ensure automated log analysis tools keep up with the demand, being capable of
efficiently handling massive volumes of logs with high accuracy. Tipping's
robustness, versatility, efficiency, and scalability make it a viable tool for
the modern automated log analysis task.