title: PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining
    Tasks
summary: Large Language Models (LLMs) have the potential to semi-automate some process
  mining (PM) analyses. While commercial models are already adequate for many
  analytics tasks, the competitive level of open-source LLMs in PM tasks is
  unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
  benchmark for PM focusing on domain knowledge (process-mining-specific and
  process-specific) and on different implementation strategies. We focus also on
  the challenges in creating such a benchmark, related to the public availability
  of the data and on evaluation biases by the LLMs. Overall, we observe that most
  of the considered LLMs can perform some process mining tasks at a satisfactory
  level, but tiny models that would run on edge devices are still inadequate. We
  also conclude that while the proposed benchmark is useful for identifying LLMs
  that are adequate for process mining tasks, further research is needed to
  overcome the evaluation biases and perform a more thorough ranking of the
  competitive LLMs.
paper_authors._id: 6699eeebfbc9ba8a122714f8
paper_authors.name: Alessandro Berti
paper_authors._id: 6699eeebfbc9ba8a122714f9
paper_authors.name: Humam Kourani
paper_authors._id: 6699eeebfbc9ba8a122714fa
paper_authors.name: Wil M. P. van der Aalst
paper_title: PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining
    Tasks
paper_summary: Large Language Models (LLMs) have the potential to semi-automate some process
  mining (PM) analyses. While commercial models are already adequate for many
  analytics tasks, the competitive level of open-source LLMs in PM tasks is
  unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
  benchmark for PM focusing on domain knowledge (process-mining-specific and
  process-specific) and on different implementation strategies. We focus also on
  the challenges in creating such a benchmark, related to the public availability
  of the data and on evaluation biases by the LLMs. Overall, we observe that most
  of the considered LLMs can perform some process mining tasks at a satisfactory
  level, but tiny models that would run on edge devices are still inadequate. We
  also conclude that while the proposed benchmark is useful for identifying LLMs
  that are adequate for process mining tasks, further research is needed to
  overcome the evaluation biases and perform a more thorough ranking of the
  competitive LLMs.
paper_ai_keywords: Large Language Models (LLMs) process mining (PM) analyses PM-LLM-Benchmark domain knowledge process-mining-specific process-specific implementation strategies public availability evaluation biases edge devices